From simulation to the SO-101
Our simulation results show that a monitor which learns from outcomes catches failures better than one kept frozen. Now we are taking it to a real robot, where conditions drift for real. The setup is built and recording.
Two of our demonstrations, played at twice real speed: front camera, side camera and the camera on the arm's wrist. A person moves a leader arm and the white follower arm copies it; the follower's cameras and joint readings are what a policy will learn from.
The setup
The robot is an SO-101, a low-cost open-source arm with six Feetech servos. A second, identical arm acts as the leader: a person moves it by hand and the follower copies each joint, which is how we record demonstrations.
Three cameras watch the work, all at 30 frames per second:
- a front camera (Logitech C920, 1280 × 720) above the table on the operator's side, looking out over the arm, so left in the image is left for the robot;
- a side camera (Logitech C270) from one corner, which sees what the front camera loses when the arm reaches in;
- a wrist camera on the gripper, which sees the object up close during the grasp.
Printed AprilTags mark the four corners of the workspace, the inside of the bowl and the top of the box. The front camera reads them, so we can tell from the images alone whether the box ended up in the bowl. That gives each episode its success label without a person judging it, which matters because SFF learns from those labels.
The task
"Put the staple box in the bowl." The box starts on one of ten taped spots on the mat, sometimes turned a little; the bowl stays in one place. We record six demonstrations per spot, 60 in all, with LeRobot. At the time of this post 25 are done.
This is a narrow task on purpose. SFF studies a policy that is trained once and then frozen, and what interests us is what happens when the world around it changes. A policy that does one thing reliably under normal conditions, and starts to fail when the light, the clutter or the objects change, is exactly what we need to watch.
Engineering the recording rig
We tuned the rig until every demonstration came in clean, at full frame rate from all three cameras.
- Three cameras on one hub. Running the front camera at 720p lets all three cameras share a single USB 2.0 hub and hold a steady 30 frames per second.
- A fast control loop. Encoding video after each episode instead of during it, with the live display off, keeps the arm control loop at a steady 30 steps per second.
- Storage. LeRobot writes every camera frame as an image before encoding, so each 40-second episode needs about 4 GB of free space while it records.
What comes next
- Finish the 60 demonstrations and fine-tune SmolVLA on them, and ACT as a second, different kind of policy.
- Freeze the policy, record a run of normal episodes, and then change conditions one at a time: a dimmer or warmer lamp, clutter on the table, a box of a new color, a nudged camera.
- Run SFF and the frozen monitor side by side on the same episodes and see whether learning from outcomes still pays on a real arm.