Early experiments: a language model driving two arms
Before we trained a single policy, we asked a bolder question: can a large language model run a two-arm robot on its own? We gave Claude Opus 5.5 the cameras and a set of motion tools, and it planned and carried out a full handoff between two arms, with no demonstrations and no training.
The left arm picks up an eraser and sets it down at a handoff spot, then the right arm picks it up and drops it in the bowl. Every move was chosen by the language model. Front and side cameras; the clip runs at 4× speed while the arms move and 16× while they wait, with captions in the video.
The setup
Two SO-101 arm pairs, one pair per hand. Claude Opus 5.5 controlled the two follower arms through a small tool server we built. Through it the model could read joint angles and motor temperatures, move a gripper tip to any point on the table (we wrote the inverse kinematics from the arm's published model), open and close each gripper, and look through four cameras: front, side and one on each wrist.
Safety lived in the server, not in the model. Each arm had its own workspace box, motion was capped at 5° per step, the grippers kept a minimum height above the table and at least 10 cm from each other, and any motor above 55 °C stopped the run. The model was free to plan; the hardware stayed protected.
The handoff
The objects sat on the left arm's side of the table and the bowl on the right, so neither arm could finish the job alone. The model worked out the plan itself:
- The left arm used its wrist camera to line up on the eraser and pick it up.
- It carried the eraser to a handoff spot between the two arms and let go.
- The right arm found the eraser there, picked it up and dropped it in the bowl.
That is a coordinated two-arm task, solved from camera images and a short instruction, with the model reasoning through every grasp and every move. One full run took about five and a half minutes.
Where it led us
Watching a language model drive the arms showed us exactly what a learned policy adds: speed and fine contact skill. A policy trained on demonstrations produces a whole chunk of smooth actions per call and learns grasps from the motion itself. It is also what our research needs. SFF reads the internal features of a frozen, learned policy such as SmolVLA and learns from its outcomes. So the next step was to record demonstrations and train one, which is where the SO-101 work picks up.