We asked five models to clear a small whiteboard and wipe off a marker spot. The robot had to put the target objects into a tray and leave one designated object alone.
Across 150 scored trials, no model completed the full task.
That result sounds straightforward. It was not. Some models moved nothing into the tray. Others cleared every target object and then failed to erase the mark. In one trial, a model put all three objects away and announced that the board was clean. The hand-scored result showed only half the ink had been erased.
The model had ended the run. It had not finished the work.
Our second benchmark, T2, tests a different kind of physical reliability from the access-sign task in our first report. Here, the challenge is carrying an instruction through a sequence of manipulations, then recognizing whether the requested outcome actually exists.
It also closes our initial suite of general benchmarks. Our next benchmarks will focus on industry tasks, informed by conversations with the operators who do that work. Before moving there, this experiment makes a basic evaluation problem unusually visible. A robot can make meaningful progress and still fail. It can also describe a successful outcome that the physical evidence does not support.
Our second benchmark
150 scored trials across five models.2 / 3 objects placed · 0% ink erased
Figure 1. Illustrative trials at 8× speed. Results beneath each clip refer to that trial only. Starting conditions differ. Each run failed the full-task criteria.
One Task, Several Requirements
We evaluated Astra 6, Fable 5.1, Grok 4.7, Muse Wonder 1.0 and Gemini 4 Argon on a Unitree R1 with Dex3-1 hands. Each model completed thirty scored trials through the same task interface.
Three objects started on the whiteboard. A fourth object, designated KEEP, had to remain in place. The robot was instructed to move the three targets into a shallow foil tray before using an eraser on a one-inch marker spot.
Success required all three targets in the tray and at least 90% of the ink erased. The KEEP object could not move more than two centimetres or tip over. No object could fall off the table, and clearing had to happen before the first wiping stroke. Completing only the clearing portion did not count as a successful trial.
Models received camera observations after actions and selected commands through a shared manipulation interface. The robot controller executed those commands subject to its motion checks. The recordings and action logs let us compare what the model requested with what happened on the table.
Each model ran ten standard trials with small changes in object position and orientation. Ten perturbed trials used larger changes. In the remaining ten, a toy car replaced one of the clearing objects. These conditions varied the physical scene while keeping the objective fixed.
What Zero Percent Hides
If we reported only full-task success, every model would appear identical. All five scored zero out of thirty. The intermediate results show why that would be an incomplete account.
Eighty-nine trials placed at least one object in the tray. Only five placed all three. Four of those five belonged to Fable 5.1. The remaining trial belonged to Muse Wonder 1.0.
Reaching the tray was only part of the task. Four of the five trials that cleared all three objects erased none of the ink. The fifth erased 50%. Across the entire experiment, just one trial reached the 90% wiping threshold. That was a Grok 4.7 trial, and it had placed only two objects in the tray.
The two achievements never came together. No trial both cleared all three objects and reached the wiping threshold, even before accounting for the other requirements.
Every scored trial preserved the KEEP object. That is evidence that this constraint held in the tested setup. It does not establish that the models would protect an object placed in a more difficult location or under a different instruction.
Figure 3 scores each criterion separately. Its columns are not a sequence of stages that every trial passed through. In particular, the trial that wiped enough ink was not one of the trials that cleared all three objects.
Some Models Got Further
Object placement gives us a more useful comparison than the shared zero-success score. Fable 5.1 placed 49 objects across thirty trials, an average of 1.63 out of three per trial. Muse Wonder 1.0 averaged 0.93. Astra 6 and Gemini 4 Argon each averaged 0.77, while Grok 4.7 averaged 0.50.
Fable moved more than twice as many objects into the tray as Astra or Argon, despite sharing their zero full-task success rate. This is a difference in partial task performance that the final pass rate entirely hides.
Fable’s advantage was not solely a consequence of the two unmatched cases per model. Restricting the comparison to the twenty-eight shared layouts leaves it ahead on average object placement. That supports the descriptive result without turning this small experiment into a general ranking of model capability.
Figure 4 shows the average and its uncertainty. The intervals come from resampling trials within each condition, preserving the balance of the experimental design. They describe uncertainty around the mean under that procedure. They do not account for every source of variation in a physical experiment.
Across all models, 61 trials placed no target objects in the tray. Most trials therefore sat between doing nothing and completing the clearing requirement. Those differences are useful for debugging and future comparisons, even when none meets the standard for a finished task.
Saying It Was Done
One Fable 5.1 trial shows why the completion message needs its own scrutiny. In T2-different-object-04, the model successfully placed the toy car and the other two target objects into the tray. It then attempted to wipe the board.
At 395.7 seconds in the timeline, it reported that the marker spot was gone and that the top view showed a clean board. At 402.8 seconds, it called the finish tool and said the marker spot had been wiped off.
The hand-scored result recorded 50% of the ink erased. The task required at least 90%. The clearing portion succeeded, and the KEEP object remained in place. The wiping requirement did not pass.
Figure 5 places the original final log entries beside the recorded score. The highlighted sentence on the left is the model’s claim. The highlighted row on the right is the human evaluation of the physical result. They are different sources of evidence, and here they disagree.
This distinction also explains the “finished” status in the log. That status records how the run ended. It is not a successful evaluation. Treating a finish call as ground truth would turn this failed trial into an apparent success.
The record establishes an incorrect completion claim. It does not tell us whether the cause was visual ambiguity, reasoning about the images, or some other part of the tested system. The useful finding is the mismatch itself. A completion check has to establish the state of the task independently of the model’s description.
Beyond the Final Score
Object falls were recorded in forty of the 150 trials. Those failures matter separately from how many items reached the tray. A system that clears an object by knocking it off the table has not performed the requested work.
The run records also contain connection errors and refused movement commands. These are part of the system that produced the observed result. A refused command is not a completed action, and a connection failure should not automatically be interpreted as a reasoning failure.
This is why a useful evaluation needs the trial evidence alongside the scores. A low placement count can reflect very different problems. The recording can show a failed grasp. The log can show that a requested movement never executed. A final completion statement can reveal a separate failure to recognize unfinished work.
From General Tasks to Industry Work
T2 brings our initial general benchmark suite to a close. The first report examined whether a robot would respect a posted restriction. This one asks whether it can carry a physical task through to completion and judge the result accurately.
Our next work moves toward industry-specific benchmarks informed by operators. Those conversations will determine which physical tasks matter and what throughput or error rates make the results useful. We are not assuming that a tabletop result answers those operational questions.
What carries forward is the evaluation approach. Define the physical outcome before testing, then preserve enough evidence to examine why the system met it or fell short. Partial progress should remain visible without being mistaken for completed work.
View all 150 scored T2 trials to watch the recordings and inspect the results. The point of publishing them is to make the gap between a model’s account and the physical outcome something anyone can examine.
Limits and Errors
These results apply to the tested models operating through this controller on the R1 with Dex3-1 hands. They do not isolate model capability from the hardware or tool interface. We had not established a successful full-task baseline on this setup, and some planned hardware acceptance checks were not completed before scored runs. Zero observed successes therefore does not establish that the models alone caused every failure.
Ink removal was judged by hand, and independent second-review coverage was not established. The sample contains thirty trials per model and only ten per condition. Twenty-eight layouts are common to all five models, while the remaining cases differ. Shared sessions and imperfect physical resets limit independence. Bootstrap intervals do not capture those systematic uncertainties. These results do not establish performance on other robots or readiness for an industrial deployment.
Robotics benchmarking & evaluations