Next: can a robot's own foresight warn of its failures?
This post is about work we have not run yet. It sets out the question, the plan and what would count as an answer, so the results can be judged against it later.
Two kinds of robot policy
Most of the robot policies we work with, such as SmolVLA, are vision-language-action models: they read camera images and an instruction and output the next few actions. A newer kind, world action models, are built on video generators and predict the next few seconds of video together with the actions. NVIDIA's DreamZero is one example; its authors report that it handles tasks it was not trained on better than current vision-language-action models.
That design has a side effect. A world action model states what it expects to see. If the arm is about to drop the box, the frames it predicted and the frames the camera records should start to disagree. DreamZero's authors report that most of its failures come from errors in the generated video.
The question
Is that disagreement a useful failure alarm?
- Does the gap between a policy's predicted future and what the camera then sees forecast failure on the same episodes where we already measure our other monitors?
- Is it better or worse than monitors that read the policy's internal features, like SFF, or one that only watches the clock?
- Does it raise false alarms when something changes that does not matter, such as the color of the walls? A video model may be surprised by a new background even when the task is going fine. Our success-only threshold was built for this case.
- Are world action models easier to monitor than vision-language-action models?
What others have done
Checking imagination against reality is not new. FFDC checks a world action model's imagined future against real observations to decide when to replan. World-Coherent Decoding uses the gap between imagined and realized futures to choose among candidate plans. CheckVLA flags deviations from an action-conditioned world model's prediction, and VLA-Corrector cuts off stale actions when the observed change does not match what the policy predicted.
These methods use the gap to act. What is missing is a careful measurement of how good it is as a forecast: against other monitors on identical episodes, across both kinds of policy, under conditions that change without warning, and on a low-cost arm. That is what we plan to measure.
The plan
- Simulation first. Run world action models and vision-language-action models on the same LIBERO tasks, with the same seeds and the same drift conditions as our SFF streams, and score every episode with every monitor.
- A go/no-go check. Before collecting the main data, register a pass criterion for the foresight signal on one test suite. If it fails, we report that and test a simpler variant: a SmolVLA trained to also predict its own future.
- The SO-101. Train a world action model and SmolVLA on the same demonstrations from our rig, run both under the same changes, and let the monitors watch without ever stopping the arm.
As with SFF, settings will be locked before any test suite is opened, and every number we report will come from a logged run.