How to evaluate a robot policy on real hardware

A robot policy evaluation should answer a question that matters to a decision: whether a candidate improves a task, whether a capability survives a changed condition, or what a model can demonstrate to a potential robot manufacturer. The design follows from that question.

This guide describes a planning method, not results from a completed Coop study. It is intended to help a model developer move from an interesting demonstration to a bounded, inspectable comparison.

Write the question before choosing the score

Name the model or versions, the intended task, and the decision the evidence should support. “Is the new checkpoint better?” is underspecified. “Does the candidate improve autonomous completion on this task while preserving the reference’s performance on these other tasks?” is a question a protocol can address.

Describe how the task relates to a workflow. State the boundaries: an object-placement test may represent one step of a larger process, but it does not establish that the whole process can be automated.

Qualify the complete configuration

Record the robot, end effector, cameras, sensing, controller, compute, and policy interface. Check observation formats, action spaces, timing requirements, and any adaptations needed to make the policy runnable. Keep that integration work separate from the scored evaluation.

The model’s development robot, your evaluation robot, and an OEM’s intended robot may all differ. A common category such as “humanoid” does not close that gap. Document differences and restrict conclusions to the configuration actually tested.

Pilot, then freeze the protocol

A pilot checks whether the task can be reset consistently, scored observably, and recorded well enough to inspect. It also reveals the real cycle time, including resets and recovery. Pilot runs are useful for design, but should not be mixed into a scored block chosen after looking at their outcomes.

Before scoring, version the protocol. Specify the trial matrix, starting-state ranges, success rules, timeouts, assistance policy, ordering, exclusion rules, and reporting plan. Keep tuning and held-out evaluation conditions separate.

Identify every policy and configuration version.
Define what success looks like without relying on intent.
Set rules for interventions, interruptions, and reruns.
Plan trial allocation and the method for describing uncertainty.

Execute a comparison that can be reconstructed

Assign a trial identifier before each attempt and connect it to the protocol, session, configuration, and condition. Where appropriate, balance policy order within sessions rather than running one model only before a calibration or environment change.

Preserve the entire attempt record. A human reset during execution, a timeout, or a recording failure should have a documented status. If a prespecified rule permits a rerun, link it to the original rather than replacing it.

Report what the evidence supports

Report outcomes by task and condition, with the number of relevant trials. Separate autonomous and assisted completion. State how failures and timeouts enter timing summaries and how uncertainty was assessed for the actual study design.

A narrow or inconclusive result can still be useful. Show where performance is established, where observations vary, and which follow-up question would reduce uncertainty. An OEM conversation should include the embodiment profile and remaining transfer questions alongside the result.