First results: can a robot's imagination warn us before it fails?

October 27, 2026. Shaurya Beriwala, Amit Jadhav

Eight days ago we wrote about an experiment we had not run yet: whether a robot that imagines its own future could warn us when it is about to fail. We have now run the first stage in simulation. The short answer is yes. The more interesting part is what we found along the way, including a much simpler warning sign we had not expected to matter.

A robot that imagines what happens next

Most robot "brains" we work with look at a camera image, read an instruction such as "put the soup can in the basket", and decide on the next few movements. A newer kind, called a world action model, does something extra. Before it moves, it imagines a short video of what the next moment should look like, and then works out the movements that would make that video come true.

What it imagined: decoded from the model — One moment from one episode. On the left, the far end of the short video the robot imagined. On the right, what its camera recorded at the end of that same stretch. Here the two are close. The warning score is how far apart they are. These two pictures come from a separate check we ran on the same robot and the same task, not from one of the attempts counted below.

What happened: the camera's own view — One moment from one episode. On the left, the far end of the short video the robot imagined. On the right, what its camera recorded at the end of that same stretch. Here the two are close. The warning score is how far apart they are. These two pictures come from a separate check we ran on the same robot and the same task, not from one of the attempts counted below.

That gives us something to check. If the robot imagines the can dropping into the basket, and a moment later the camera shows the can still sitting on the table, something has gone wrong. Our question was whether that mismatch between imagination and reality is a useful early warning, one that could let a person or a safety system step in before a task fails.

What we ran

We used a publicly released world action model, FastWAM, on LIBERO-10, a standard set of ten simulated household tasks such as putting two groceries in a basket or turning on a stove and placing a pot on it.

We ran 1,000 attempts. Some were under normal conditions. The rest were under 19 kinds of change that nobody tells the robot about: dimmer or warmer lighting, objects painted a different color, a camera nudged a few centimeters and tilted a few degrees, slippery fingertips, weaker motors, a new background color, or noisy camera images. 870 of the 1,000 attempts succeeded.

Normal: level L00 — The same starting scene of one of the ten tasks, as the robot's camera sees it, under four of the changes we used. The robot is given no warning that anything has changed, and no way to tell which condition it is in.

Dim light: light x0.25 — The same starting scene of one of the ten tasks, as the robot's camera sees it, under four of the changes we used. The robot is given no warning that anything has changed, and no way to tell which condition it is in.

Each time the robot plans its next movements, it imagines the coming moment. Less than a second later, we compare that imagined moment with what the camera actually recorded, and turn the difference into a warning score. We make the comparison in the model's own compact summary of an image rather than pixel by pixel, because that is the form in which the model does its imagining.

A pass mark written down before we looked

We use one main measure throughout. Pick one failed attempt and one successful attempt at the same task, at random. How often does the warning score rate the failure as more worrying? A score of 0.5 is a coin flip. 1.0 would be perfect.

Before looking at any results, we committed our pass mark to a time-stamped record: at least 0.70, and at least 0.05 better than a monitor that only watches the clock. The clock check matters because failed attempts tend to run longer than successful ones, so a timer can look clever. We cut every attempt off at the length of the shortest one, so a timer scores exactly 0.5.

We registered two ways of measuring the gap between imagination and reality. The first asks how far apart the imagined and real moments are. It scored 0.765 and passed. The second asks only whether they changed in the same direction, ignoring how large the change was. It scored 0.565 and did not pass.

The twist: a robot that stops moving

One of our checks was meant to be a formality. It asked a much simpler question with no imagination involved: how much did the camera view change across a window of a fifth of a second, or of two fifths? We expected it to be useless as a warning.

It was not useless. It was strongly informative, just the opposite way round from what we assumed. Failing attempts are often ones where the robot stalls, hesitates or gets stuck, so a scene that has stopped changing is itself a warning sign. Read the right way round, "is the scene standing still?" scored 0.753, almost as high as the imagination signal, with no world model at all. We first misread this check in our own notes and have corrected them.

Each signal's score on the same 900 attempts (the remaining 100 were used to set the monitors up). The stillness score is the best of four readings, from two camera views over two short windows. The internal-signals monitor reads the robot's own internal activity, the kind of input our SFF monitor uses, and here it ran with default settings.

A single pair of attempts shows how much room there is inside those numbers.

Two attempts at the same task, each picked as the middle of its group rather than the clearest. The line is the highest warning score reached so far, over the stretch we score, the first twelve and a half seconds. The two finish close together. The score tells failures from successes across the whole set of attempts, not reliably within any one pair.

So we asked the harder question: does the robot's imagination tell us anything that stillness does not?

To find out, we combined the two signals and compared the combination with stillness alone. Each combination was always scored on attempts it had not been fitted to. Adding the imagination signal improved the score by about 0.08 on the same scale, and the improvement held when we also added the robot's internal signals.

There was one more surprise. Once stillness is taken into account, the way of measuring the imagination gap that failed on its own helps too. Which of the two helps more depends on how the numbers are prepared before the comparison, so we are not claiming a winner between them. Our best explanation is that imagination errors are naturally larger when a lot is moving, and a lot moves in attempts that are going well. Once you allow for how much the scene is moving, the surprise that remains points toward failure.

What we are not claiming yet

This is simulation, one policy and one set of tasks. Nothing here has run on a real robot.
The stillness finding came after we saw the first results. Something found while looking around is a hypothesis, not a conclusion, however good the numbers look. The pass mark above was set in advance; the comparisons with stillness were not.
Our uncertainty was larger than we first reported. We found and fixed a mistake in how we computed our error bars: attempts that started from the same scene were being counted as if they were unrelated. The fix does not move any score and does not change the pass. It does widen the uncertainty: for the passing measure, the range we are 95% confident about is now 0.699 to 0.827.
The internal-signals monitor has not been tuned for this robot. We will not say it does worse than stillness until it has had a fair setup.

What comes next

We have written down a confirmation test before collecting any new data. The exact way of combining the signals has been fitted on the 900 scored attempts and then frozen, and its fingerprint is recorded so it cannot be quietly changed. A fresh batch of 696 to 900 attempts will then decide one question: does adding imagination to stillness still help, by a margin whose uncertainty range stays above zero? Those attempts run under six of these conditions, starting from the scenes the simulator sets aside for development, the fifty we have never used first.

If it does, we will have a result we trust. If it does not, we will report that just as plainly. We also still want to see this on a real arm. Our plans there have changed since the last post, and we will write about them separately. Our other monitor, the one that keeps learning from how each run ended, has already been through that step: we tested it on recorded runs of real arms, and wrote that up here.

As before, every number in this post comes from a logged run.