“The robot completed the task” leaves several questions open. Was there a timeout? Did someone reposition an object? Was this the first attempt or a rerun? A useful metric makes those distinctions explicit.
Measurement starts with the protocol and ends with the trial ledger. The definitions below are a practical way to structure that record; each evaluation still needs rules fitted to its actual task.
Define success in observable terms
Specify the required end state and any constraints that must hold throughout execution. Identify who or what scores it, when the observation is made, and what happens when the evidence is ambiguous. Set the timeout before testing.
Separate task outcome from execution status. A trial can be scheduled but never started, interrupted by infrastructure, or completed with a task failure. Those are different records and may enter different metrics according to prespecified rules.
Separate autonomous and assisted completion
Define which human actions count as an intervention. Examples could include repositioning an object during execution, issuing an extra instruction, or manually recovering the robot. Record the type, timing, and reason for each intervention.
An assisted completion may be informative, but it should not count as an autonomous success. Report the number of trials with any intervention separately from the total number of interventions: one trial can require several forms of help.
Make the denominator reconstructable
For an autonomous success rate, report the number of autonomously successful trials and the number of eligible attempted trials under the stated inclusion rules. Keep scheduled trials and non-starts visible in the ledger, even if they do not belong in that denominator.
Infrastructure exclusions need a reason and a rule defined in advance. Record exclusions and reruns alongside the reported rate so a reader can understand what was omitted. Do not selectively remove difficult attempts as setup problems after seeing the model’s outcome.
Report timing without hiding failures
Define when timing starts and stops. Distinguish policy execution time from preparation, reset, or recovery time where relevant. A completed task, a timeout, and an interrupted attempt should not become indistinguishable timing values.
If a completion-time summary includes only successful trials, label it that way and report the failure and timeout counts beside it. A faster average over a small successful subset does not, on its own, establish a better policy.
Match uncertainty to the evaluation design
A percentage without a trial count is hard to assess. Report counts by task and condition and describe the uncertainty method. Repeated runs in the same session can share calibration, operator, or environmental effects, so treating every run as an independent observation may be inappropriate.
Plan the trial allocation and analysis for the decision being made. A small observed difference may remain inconclusive. If it warrants confirmation, define a follow-up study rather than adding trials selectively until one model appears to win.
Connect the metric to the evidence
Use stable trial identifiers to connect scores, intervention events, execution logs, and recordings. Keep a scoring note for ambiguous cases and preserve the original record when a correction is made.
The goal is a result that another person can interrogate: what was attempted, what happened, how it was scored, and what the evidence does not establish. That record is especially useful when comparing model releases and investigating a regression.
Put the method to work
Independent robot benchmarks, grounded in real work