On real robots, the failure monitor that keeps learning wins

October 28, 2026. Amit Jadhav, Shaurya Beriwala

We tested SFF, our failure monitor that keeps learning after deployment, on real robot arms run by four different robot AI models. On all four, when the robot's job changed without warning, SFF told failing runs from successful ones better than a frozen monitor and better than the standard detect-and-reset approach.

4 of 4 robot models: SFF won on SmolVLA, pi0.5, pi0 and MolmoAct 2, each time with a 95% interval above zero.
+0.12 AUROC over the frozen monitor, and +0.11 over detect-and-reset, with the four models weighted equally.
Same answer as simulation, where SFF also handled dim light, moved cameras and worn grippers.

SFF (keeps learning)Frozen monitorDetect and reset

Real SO-101 runs from ArmnetBench, after the robot switched to a new task without warning. Each bar is a monitor's within-task AUROC: the chance that a failing run gets a higher risk score than a successful run of the same task. Every monitor read exactly the same signal from the robot's AI model.

Why this matters

A robot that works in a lab will meet new jobs, new objects and new conditions in the field. A failure monitor is supposed to warn before a run fails. Most monitors are trained once and then frozen, so the moment the robot's situation changes, their warnings stop meaning much, and nobody notices. SFF keeps learning from the one thing every run reports for free: whether it worked.

Frozen monitors break when the job changes

SFF (keeps learning) Frozen monitordashed line: coin flip (0.5)

Ranking quality before and after the task switch. Before it, the frozen monitor was fitted to the job and often looks fine. After it, the frozen monitor drops toward a coin flip, while SFF keeps learning and ends ahead on all four models.

On pi0.5 the frozen monitor was the best method before the switch (0.723) and fell to 0.552 after it, while SFF held steady. On MolmoAct 2 the frozen monitor fell from 0.608 to 0.517, close to a coin flip, while SFF reached 0.666. On pi0, SFF starts at a coin flip because it has learned nothing yet, then climbs to 0.688 as outcomes come in. Its lead keeps growing after a change: on pi0, +0.067 over the first 20 runs and +0.169 after that. Detect-and-reset, the usual fix for changing conditions, did not help: on pi0.5, pi0 and MolmoAct 2 its change detector never fired.

Tested on real robot runs

Real SO-101 arm: Ring onto a peg — Five of the real tasks, from the robot's front camera (ArmnetBench, Apache-2.0).

Real SO-101 arm: Stack blocks — Five of the real tasks, from the robot's front camera (ArmnetBench, Apache-2.0).

We used ArmnetBench, a public set of 2,499 runs of real SO-101 arms, each marked by a person as a success or a failure. For four AI models the benchmark also published the exact trained models that drove the arms. SFF reads signals from inside the robot's AI model, so we fed every recorded run back through the model that ran it and checked the result: the replayed model reproduces the actions the real robot took to within about one degree. One MolmoAct 2 model missed our replay check, so we left it out.

It also handles physical changes

Nominal: as trained — The physical changes from our simulation study: dim light, recolored objects, a moved camera, worn gripper fingers, two at once, and a harmless change of wall color.

Dim light: light ×0.25 — The physical changes from our simulation study: dim light, recolored objects, a moved camera, worn gripper fingers, two at once, and a harmless change of wall color.

The real-robot runs test a change of job. In simulation we changed the physical world instead, and SFF ranked failures better than the frozen monitor on 4 of 5 test suites, including the one we registered in advance, and beat detect-and-reset on all of them. Read the simulation results.

How SFF works, in two sentences

SFF starts from an ordinary failure monitor trained on normal runs, then, after every run, refits its last layer from whether the run succeeded, with constant memory and no stored data. Its alarm threshold moves only after successful runs, which keeps false alarms at the rate the operator asks for.

How we kept it honest

The test plan for each robot model was written down and committed before any of its results existed.
SFF's settings were fixed in simulation and never tuned on these robots.
A monitor that only watches the clock scores 0.5 everywhere, so no method wins by timing runs.
A simpler monitor that also learns online comes close to SFF (+0.029 behind it across the four models): learning during deployment is what matters most.
The data is small, 434 real runs in all, and the streams reuse the same runs, so these results back up the simulation study rather than replace it.

What comes next

On MolmoAct 2 we also tried a new kind of change: an update to the robot's own model on the same job. SFF was ahead there too, but with only 10 test segments that result is not yet conclusive. And we have trained pi0.5 on our own SO-101 demonstrations, so the next test runs live on our arm. Read about our SO-101 setup.