The 99% Problem: Why Robot Benchmarks Stopped Telling Us Anything

Simulation was never going to be the whole answer

Simulators are essential. They're fast, cheap, parallelizable, and safe. You can run ten thousand trials before lunch and never break a gripper. That's a real advantage, and no serious evaluation strategy should throw it away.

But a simulator is a model of the world, not the world. Every simulator makes approximations in rendering, contact physics, and actuation, and those approximations compound. A recent Annual Review of Control, Robotics, and Autonomous Systems survey on the "reality gap" breaks this down into perception mismatches (lighting, texture, sensor noise) and dynamics mismatches (friction, compliance, actuator latency, backlash), and notes that closing this gap remains one of the most pressing open challenges in robotics, despite years of work on domain randomization and system identification.

Some of these gaps are shrinking. Photorealistic rendering and better contact solvers help a lot. But a related survey on embodied manipulation data points out that the harder gaps, the ones involving torque saturation, compliance, and multi-body contact effects, are only partially captured by even the best current engines, because many pipelines still simplify contact dynamics or use coarse timesteps to stay fast enough to train on (Data Pyramid for Embodied Manipulation). A policy can learn to exploit exactly those simplifications. It gets very good at the simulator's version of "picking up a cup" without ever encountering what a cup actually does when your gripper is a millimeter off.

This is why a model's simulated score and its real-world score aren't just offset by some fixed amount you can mentally subtract. They can rank models differently. A checkpoint that looks best in sim isn't guaranteed to be the one that works best in your warehouse.

Today's benchmarks have a saturation problem

Even setting the sim-to-real gap aside, there's a second issue, one visible from the leaderboards themselves.

Most robotics benchmarks ship with a fixed set of tasks. Fixed task sets get solved. NVIDIA’s research team identifies infrequently refreshed tasks as a reason scores reach a ceiling and comparisons become less informative. The caption accompanying its benchmark-saturation figure states: “Almost every model paper reports results on this benchmark, but the saturation as shown makes it difficult to extract meaningful conclusions about model performance.” (NVIDIA Technical Blog).

LIBERO, one of the field's most widely cited suites, is a good example. Multiple recent papers now note that state-of-the-art models approach or exceed 99% success on it, which sounds like victory but mostly means the benchmark stopped being able to tell models apart.

The more concerning finding is what happens when you push on that number. LIBERO-PRO, a 2025 evaluation, took standard LIBERO tasks and applied small, reasonable perturbations: swapping an object for a similar one, moving its start position, or slightly rewording the instruction. Models that scored over 90% on the standard benchmark collapsed to 0% success under these perturbations. The paper's conclusion is blunt: high scores mostly reflected memorization of scene layouts and action sequences, not task understanding. The robots weren't performing a task. They were replaying one.

That's the real cost of saturation. It's not just that scores stop moving. It's that a benchmark can keep producing impressive numbers while measuring something closer to rote recall than capability.

Real-world variation is the actual test

Real environments vary in lighting, clutter, object identity, and surface friction. A single setup captures only a slice of those conditions: a study of vision-language-action models found meaningful performance differences across tasks, environments, and robot platforms (Benchmarking VLA Models on Robotic Learning Tasks).

In a deterministic setup, the same starting state and actions produce the same outcome. Stochastic evaluation introduces randomness, such as sampling object positions or sensor noise; simulation can do this too. A policy trained on narrow, repeated conditions may rely on a cup always appearing in one place, then struggle when that expectation no longer holds.

The goal is relevant variation, not randomness for its own sake. We can sample starting conditions from declared ranges, keep the task and success criteria fixed, and compare models on the same saved cases. Pairing those controlled comparisons with physical testing helps reveal whether performance holds beyond familiar benchmark conditions, a concern also motivating work such as REALM.

What this means for evaluation

None of this is an argument against simulation, and it's not a claim that testing on real hardware automatically fixes it either. Real-world evaluation brings its own headaches: it's slower, more expensive, harder to reproduce exactly, and every trial has to be run and reset by hand. A single real robot rollout tells you less in isolation than a thousand simulated ones do. Simulation and real testing aren't competitors. They answer different questions, and only one of them can tell you what happens when the gripper actually closes on the actual object in actual light.

What the field needs, and what we think is currently missing, is evaluation that treats the robot itself as the ground truth. That means testing on real, documented hardware configurations. It means defining tasks around what industry operators actually want automated, not what's convenient to simulate. It means tracking failures, not just success rates, because a 74% success rate that fails the same way every time tells you something a 74% that fails randomly does not. And it means being honest that a benchmark's job isn't to make a model look good. It's to tell you, accurately, whether it's ready.

We're building Coop around that idea, starting with a single documented robot, the Unitree R1, and a small set of real tasks. We don't think we've solved this problem. We think it's worth working on in the open, with results you can actually go watch.