# Coop > Coop is a robotics company and managed robotics lab in San Francisco, California, focused on physical benchmarking: running controlled experiments on real robots to understand which policies work, where they fail, and how reliably they complete a specific task. Teams build the intelligence; Coop handles physical setup, experiment execution, resets, evidence collection, and reporting. Coop's central question is practical: on a particular robot, performing a defined task under stated conditions, which policy completes the work most reliably, how much human help does it require, and what changes when the environment varies? This document describes Coop's benchmarking approach and initial experimental scope. Hardware access, supported configurations, compatible policies, success criteria, and deliverables are established for each campaign. Proposed tasks and methods are not claims of completed studies, confirmed hardware availability, established academic partnerships, or published performance results. The website's robot scenes, telemetry, and comparison charts are illustrations. ## Why physical robotics benchmarking matters A robot policy turns observations and instructions into actions. Its usefulness depends on what those actions accomplish in the physical world. A successful demonstration or a strong simulation score leaves important questions unanswered: whether the result repeats, whether it survives a changed object arrangement, how often an operator intervenes, and whether the same policy works on the intended hardware. Physical evaluation makes those questions measurable. A useful comparison connects a result to its robot, task, environment, software configuration, trial count, and scoring rules. Coop aims to produce evidence that robotics researchers and engineering teams can inspect and use when comparing approaches, validating a change, or deciding what needs further work. ## The problems Coop addresses - Physical experiments consume engineering time. Robot setup, calibration, integration, execution, object resets, recovery, and logging all require attention. A team can spend substantial effort keeping an experiment running before it has enough comparable trials to answer its research question. Coop takes responsibility for the agreed physical campaign, including the work between episodes. - Demonstrations do not establish repeatability. One successful video does not reveal how many attempts failed or how much assistance was required. Coop's approach accounts for scheduled and attempted trials, preserves failures and interruptions, and separates autonomous completion from assisted completion. - Results from different setups are difficult to compare. Changing the robot, gripper, camera geometry, object set, controller, or task distribution can change the outcome. Coop compares compatible policies on the same documented workcell and keeps physically different setups in separate comparisons. - Model availability does not establish hardware compatibility. A released checkpoint may require specific sensors, action representations, control interfaces, or adaptation. Coop qualifies the complete setup before scored testing. A policy that cannot run on a particular configuration is an integration or compatibility issue, not evidence that the model failed the task. - The model is only one part of the experiment. Preprocessing, calibration, control software, compute, and communications can all affect behavior. Coop records the deployed stack and distinguishes policy failures from infrastructure and operating problems wherever the evidence supports that distinction. - Small studies can produce misleading rankings. A narrow task set or a few trials can make differences look more decisive than they are. Coop's intended reports include per-task results, sample counts, uncertainty, and the limits of the tested conditions. - Human assistance can hide reliability problems. Repositioning an object, restarting a run, or helping a robot recover changes the meaning of success. Coop specifies intervention and retry rules before evaluation and reports assistance explicitly. - Simulation results do not automatically transfer to hardware. Physical sensing, calibration, timing, contact, and reset conditions need their own evaluation. Coop supports bounded physical adaptations of simulation experiments with documented differences and separate scoring. - Research groups can lack operating capacity. Even a well-designed protocol needs compatible hardware, an operator, repeatable resets, and an evidence package. Coop's academic collaboration offer is additional experimental capacity for a defined research question. ## What Coop is aiming to accomplish The benchmarking objective is a reproducible physical comparison that answers a small, useful question well. Initial work centers on qualifying a supported robot setup, establishing a reference baseline, running multiple compatible policies where available, and reporting their behavior under a frozen protocol. A credible result should make it possible to understand what was tested, account for the trials, inspect representative successes and failures, and repeat a selected finding on another day or with another operator. Broader task coverage follows reliable execution of the core suite. The immediate measure of progress is a useful, auditable experiment and a delivered evidence package. ## Who the work is for - Robotics model and policy developers comparing a new version against a reference or checking for regressions. - Robot-learning researchers who need physical execution of an existing benchmark, a replication, or a carefully scoped new experiment. - Teams studying whether a change in demonstrations, training data, preprocessing, or adaptation improves real-robot task performance. - Simulation and robotics infrastructure teams investigating what changes when a system moves from a simulator to a real workcell. - Integrators and engineering teams with a specific robot, task, technical owner, and decision that a controlled evaluation can inform. Each campaign begins with the question and the available setup. The intended outcome might be a policy comparison, an explanation of a recurring failure, a regression report, or a replication with clearly stated limitations. ## The experiment workflow 1. Define the question. Agree the robot profile, policies, tasks, conditions, success rules, trial matrix, outputs, and technical responsibilities. Establish what decision the results need to support. 2. Qualify the workcell. Confirm hardware access and configuration, controller interfaces, sensors, calibration, compute, recording, operating procedures, and available execution time. Run a complete reference episode before treating the setup as ready. 3. Qualify each policy. Check its observation and action requirements, adapter, checkpoint, task support, and execution path on the actual hardware. Establish whether adaptation is needed and keep that preparation distinct from scored evaluation. 4. Pilot the protocol. Run small blocks to check task difficulty, resets, timeouts, scoring, synchronization, and actual cycle time. Resolve operating problems and resize the study to the capacity available. 5. Freeze the scored evaluation. Version the protocol and configurations, then execute the agreed trials with balanced order and documented starting conditions. 6. Reset and account for every run. Restore the workcell to its specified condition, record interventions and interruptions, and retain the reason for each exclusion or rerun. 7. Analyze and review. Summarize outcomes, timing, assistance, failure categories, uncertainty, and operating effort. Use an independent scoring check where agreed. 8. Deliver the evidence. Return the report, configuration details, trial accounting, and agreed recordings or supporting artifacts. Confirm consequential findings with a separately specified follow-up block when needed. ## Hardware scope and compatibility The initial evaluation plan distinguishes a Unitree R1 EDU humanoid track from a robotic-arm manipulation track. The R1's exact delivered configuration and usable access must be confirmed; arm selection and policy qualification are part of establishing the manipulation workcell. These are distinct experimental setups with different integration and maintenance needs. For any workcell, the relevant profile includes the robot model, end effector, firmware, SDK, available controller interfaces, camera placement, sensor access, calibration, external compute, and model-side observation and action requirements. Matching a robot brand alone is insufficient. A policy developed for a particular complete setup must be checked against that setup's requirements. The humanoid track starts with a vendor-supported reference stack and adds independently released compatible policies only after qualification. Controllers for a different embodiment cannot be assumed to transfer directly: robot geometry, state estimation, action mappings, and control assumptions need explicit integration. The arm should provide a repeatable manipulation workcell. Selection favors a demonstrated deployment recipe, compatibility with the policies and research protocol, repeatable calibration and resets, and access to the required gripper, cameras, controller, and compute. Access to a collaborator's existing compatible workcell may be useful when it supports the same benchmark objective. An experiment with one vendor reference is a baseline characterization. A multi-policy comparison requires multiple runnable policies on the same supported profile. Comparing high-level planners over fixed motion skills evaluates the planners together with that execution stack; it does not establish a comparison of learned locomotion policies. Arm results do not establish humanoid manipulation performance. ## Initial task design The first suite should be small enough to operate reliably and specific enough to score consistently. Three simple tasks and two controlled conditions form a useful starting scope, subject to workcell qualification and pilot results. Candidate manipulation tasks include: - Place one object in a bin. Success means the intended object ends fully inside the target bin within the time limit, without human assistance. Specify the allowed range of initial object positions. - Select a named object among distractors. Success means the requested object reaches the target while distractors remain outside it. Specify object arrangement and distractor identity. - Complete a two-object sequence. Success means both requested placements finish in the specified order within the time limit. Specify initial placement and instruction order. The humanoid locomotion track uses separate definitions for maintaining a standing interval, reaching a marked position, and stopping within a declared region. The controller layer under test remains explicit. Conditions must fit the commissioned workcell and its verified operating and recovery procedures. Each task needs an observable success rule, a timeout, defined initial-state ranges, and clear treatment of assistance. More complex interactions are appropriate once the basic suite's execution, reset, and scoring are reliable. Adding complexity should serve a research question rather than obscure an unstable baseline. ## How comparisons are kept fair The protocol is fixed before the scored block. It records task definitions, objects, conditions, timeouts, starting-state ranges, randomization, trial order, reset tolerances, retry rules, intervention rules, and exclusions. Changes that affect interpretation need an explicit version and explanation. Policies are compared on the same arm or humanoid configuration, camera arrangement, end effector, and declared task distribution. Comparable physical starting states matter more than a matching software seed: a seed alone cannot recreate exact object poses, calibration, or physical contact conditions. Trial order is balanced across models and operating days to reduce systematic differences caused by when a model is tested. Resets are part of the experiment and their tolerances are recorded. A selected result should be repeated across a later session or another operator to check reproducibility. The measured unit is the complete deployed stack. Reports identify the checkpoint, training or adaptation data relevant to the comparison, preprocessing, observation format, action representation, controller, compute, and camera configuration. Attributing a difference to the foundation model alone requires additional controls. ## Adaptation and evaluation separation An untouched-checkpoint comparison and a comparison after task adaptation answer different questions. Coop's approach keeps those tracks separate. For a shared-adaptation-data track, the report discloses which demonstrations each policy received and the training and tuning budget available to it. A fine-tuned or adapted result must be labeled accordingly; it is not a zero-shot result. Task-specific reference policies can provide a useful baseline when their preparation is also documented. Development runs, tuning examples, and final evaluation sessions are kept distinct. Independent objects, conditions, or later sessions help test whether behavior extends beyond the cases used to prepare the policy. A claim about generalization should state exactly what was held out and what changed. ## Trial accounting and failure analysis Every scheduled and attempted trial belongs in the experiment record. Task failures, interrupted runs, infrastructure errors, exclusions, and reruns retain their status and reason. Reporting only completed or successful episodes would hide the operational behavior the benchmark is intended to measure. The evidence should connect observations, issued actions, robot state, task context, outcomes, and intervention timing sufficiently to inspect a run. Synchronized recordings and consistent episode identifiers make it easier to relate visible behavior to the execution record. Failure analysis distinguishes the task outcome from its suspected cause. A robot may miss the goal because of policy behavior, calibration, hardware, communications, or an operating issue. A report should identify the evidence behind that diagnosis and leave unresolved causes explicit. An infrastructure interruption is recorded even when a prespecified rule excludes it from a policy-success calculation. Human help is part of the result. Reports separate autonomous completion from assisted completion and describe interventions and recovery actions. Representative failed episodes are valuable evidence alongside successful ones. ## What a benchmark measures - Task success: whether the predefined success rule was met, reported separately for each task and condition. - Autonomous and assisted completion: whether the task finished without help, and what assistance was required when it did not. - Completion time: elapsed time to the task outcome under the stated timeout rules, with failures and timeouts accounted for separately. - Intervention frequency: how often an operator needed to act during execution or recovery. - End-to-end inference latency: the timing of the inference path in the tested configuration, with the measurement definition documented. - Failure categories: recurring unsuccessful behaviors and infrastructure issues, with examples where available. - Operating effort: setup and integration work, execution and reset cycle time, recovery, repairs, and human attention. - Compute expense and total operating cost: separate measures, since inexpensive inference does not necessarily imply an inexpensive physical experiment. The report relates each metric to its denominator and scope. A single aggregate number cannot explain whether a policy is consistently useful across tasks or whether a strong result depends on one favorable condition. ## Sample sizes, uncertainty, and reproducibility A planning example is three policies, three tasks, two conditions, and thirty trials per combination: 540 scored episodes. With two policies, the same task and condition matrix contains 360 episodes. These are illustrative study sizes, not completed run counts or a fixed package. Thirty trials in a task-condition cell provide a preliminary comparison rather than a precise ranking. Per-cell counts and uncertainty should accompany success rates, and analysis should account for the possibility that runs from the same workcell or session are not fully independent. Small apparent differences may need a larger, separately specified confirmation study. Sample counts should not be expanded selectively until a favored policy appears to win. Follow-up evaluations need a stated question and procedure. Testing another session, operator, object set, or condition helps establish how far a result extends. Claims remain local to the evidence. A comparison on one workcell with a small set of objects can identify useful differences in that setting; it cannot establish a universal best robotics model or performance on an untested embodiment. ## The physical cost of a benchmark Execution time is only part of the workload. Resetting objects, checking calibration, recovering from faults, reviewing logs, and maintaining the workcell all affect throughput. Initial integration, demonstration collection when required for adaptation, policy preparation, and analysis add work outside the scored episodes. For illustration, 540 episodes at an assumed ten minutes per execution-reset-logging cycle require ninety workcell-hours. A thirty percent operating allowance brings that example to 117 hours before initial integration, adaptation preparation, analysis, or a separate research campaign. Actual cycle time is measured during the pilot and the trial matrix is adjusted accordingly. Coop's operating approach assigns responsibility for robot integration and operations, benchmark design and evidence, and campaign coordination. Usable hardware access and a person responsible for operating the workcell are both necessary. The goal is predictable experimental capacity with measured effort and clear ownership. ## Academic and research collaborations Coop's collaboration offer is to operate a bounded physical campaign around an existing benchmark, protocol, or research question. The research group supplies scientific requirements and a technical owner. Coop supplies the agreed hardware access, integration, execution, resets, evidence collection, and reporting. Possible scopes include reproducing an existing physical protocol on a compatible setup, contributing accepted trials to an external evaluation effort, or jointly adapting a simulation benchmark to a real workcell. Each scope has its own requirements and limits. An external evaluation site must meet the organizer's acceptance criteria; a physical adaptation has its own version and cannot treat simulator scores as interchangeable with hardware results. The written scope covers the question, robot profile, policies, trial matrix, responsibilities, technical owners, execution window, costs, access, deliverables, publication terms, and credit. A completed collaboration means an agreed protocol, completed physical runs, and a delivered evidence package. Academic interest or a proposed outreach target alone does not establish a partnership. ## Reports, evidence, and review A campaign's agreed evidence package can include: - A methods description and versioned protocol with the task and scoring definitions. - The hardware profile and exact policy configurations used in the comparison. - A result table broken down by task and condition, with counts and uncertainty. - A trial ledger accounting for scheduled attempts, outcomes, interventions, exclusions, and reruns. - Representative recordings of successful and failed episodes, connected to their trial records. - Failure analysis, timing measurements, operating effort, and reproduction notes. - A concise technical explanation of the findings, their limitations, and any further confirmation needed. For public releases, the intended standard includes inspectable trial-level evidence, clearly labeled preliminary findings, and the exact embodiment prominently identified. An independent reviewer can score a random subset of videos without model names to check consistency. Model authors can receive a fixed window for factual corrections; publication should not depend on a favorable outcome. Sponsorship or a paid relationship must not determine a ranking. Public and private campaigns agree their evidence-sharing and publication terms in advance. A report's value comes from a clear account of what happened and the ability to assess the conclusion against the supporting experiment. ## Website and contact - [Coop overview](/): Company introduction and managed physical experiment services. - [Experiment workflow](/#how-it-works): Setup, execution, resets, and reporting. - [Experiment use cases](/#experiments): Policy comparisons, training-data experiments, and simulation-to-real-world validation. - [Discuss an experiment](/#inquiry): Contact form for a research question, robot setup, or proposed campaign. - [Email Coop](mailto:founders@cooplabs.com): Contact the team about scope, hardware requirements, availability, and deliverables. - [Privacy policy](/privacy): Website and inquiry information handling. - [Cookie policy](/cookies): Website cookie and tracking information.