A better alarm is not always a better forecast
A failure monitor can look better for the wrong reasons. A threshold change can cut false alarms without making the scores any more useful, and a score that simply tracks elapsed time can look like a forecast. So we measure three things separately.
Does it tell failures from successes?
A monitor's score should be higher in episodes that will fail than in episodes that will succeed. We measure this as a ranking: the chance that a failing episode gets a higher peak score than a successful one. We compare episodes of the same task only, so a monitor cannot score well just by recognizing that some tasks are harder than others.
Time is the trap. Successful episodes often finish early, while failing ones run until the time limit. A "monitor" that only counts elapsed seconds would flag nearly every failure. On LIBERO-Spatial it flags failures at a rate of 1.000 and successes at only 0.101, yet its ranking is exactly 0.500, chance, once we compare scores only up to the length of the task's shortest success. That cutoff is what keeps a timer from passing as a forecaster.
How often does it raise a false alarm?
An alarm is false only in an episode that succeeded, so we measure the false-alarm rate on successful episodes against the rate the operator chose. This is a different question from ranking: moving the threshold changes how often a monitor alarms, but it cannot make the score separate failures from successes any better. SFF learns both, with one part for each, and we report them separately.
Did the warning come early enough?
An alarm after a failure has already happened is detection, not a forecast. Measuring warning time properly needs an independent record of when each failure began, so we are building one: on the real arm, AprilTags and joint readings are set up to mark the moment a grasp or placement goes wrong. That turns lead time into a number we can report with the same care as ranking and false alarms.
Numbers you can trust
Every result we publish comes from a logged run, with its task, stream, episode count and protocol attached. Our evaluation settings are locked before a test suite is opened, and the primary test is registered in advance. When we say a monitor is better, it is better under rules we fixed before we looked.