Notes on Sunday robotics

Watching back the ACT-1 foundation model release video, I wanted to get a better grasp on the state of robotics. While starting way back at open-loop systems or reading up on transformers might be sensible, I chose instead to work my way from near the top and decided to watch Sunday Robotics' demo video from 9 months ago.

Some notes: The titular claim is that the model was trained on zero robot data, only their UMI data collection gloves or human data. This was some time ago now and I wonder how much simulation plays a role now. Zero robot data is obviously very scalable. It means they use humans wearing sensor gloves while doing tasks to generate training data, which is then mapped to the robot. That is, as opposed to humans teleoperating the robot to generate training data for it. It's based on UMI (on my reading list).

memo's hand

Memo's grasping is very human-like. Having 3 grasping pockets is a keystone feature for manipulation in primates, with our opposable thumb doing the heavy lifting. When I attempted multi-object grasping, what I realized is that my thumb always acts as the pincher, grasping the object alongside one of my other 4 fingers. When I want to pick up a second or third object, I consolidate my grasp of the past object in my hand and then perform another pinch maneuver with my thumb. Key difference from Generalist and Pi: Not a parallel gripper but a 3-finger "gloved" hand, similar to BD's new Atlas robot (and RAI Institute is moving into UMI-style capture, interesting dive for another day).

Memo's 4-DOF hand (two fingers with two DOFs each and a fixed thumb). Super cool to see this telescopic base in action. I'll look more into the hardware someday.

zero-shot generalization

Zero-shot generalization: the policy performs a task it never saw in training or demonstrations. Very curious how this worked back then, since in light of Generalist's GEN-1.5 the model could infer a missing task from the object in front of it. How is a prompt even defined for robotics? Text, visual?

The model was still pretty slow, most videos are in 4x. I remember Tony mentioning once in another interview that deployment is also slowed down for safety reasons. Curious what the maximal speed of the model could be, but it also makes sense that at higher speeds a dexterous task is harder to control, even for humans.

the data collection device

Sunday's key insight was obviously their data collection device. The operators wear the data-glove as well as an ego-camera on their head.

From their ACT-1 blog post: data is the bottleneck, robotics does not have the internet to train on. They frame the data problem as an embodiment mismatch problem and co-design the capture device with the human and the robot in mind, same geometry and sensor layout. I have used teleop a bunch of times and it is a compromise on the human's side to have to conform to the robot's form. The rest of the mismatch (the forearm, the variability of human bodies) is handled by what they call skill transform, which generalizes the visual and kinematic observations so the data transfers. That is their key product, claimed 90% success rate. For navigation they condition the model on 3D maps of the home during training, so it learns to read maps instead of memorizing houses.

The gloves cost roughly $400 a pair, according to Zhao in WIRED.

Very cool look into Memo's insides, had not seen this yet. Things to take from here are the kinematics and actuator choice.

Zhao worked on Mobile ALOHA at Stanford, Cheng Chi (the CTO) on UMI. During my trip to Stanford in July I visited the exact kitchen where Mobile ALOHA was deployed! Was a very cool moment, thanks, Lukas, for showing me around (pictured below).

A thought I had for when these robots get deployed in homes: There is an issue in moving from dirty tasks (cleaning the floor, emptying the bin) to clean ones (handling food or being near people). Potential solution could be having its appendages covered in washable skin/cover or something like Loki Robotics, where they tool-change, but way more complex.

Nice visual design of Memo.

The glove solves the action side, but the forearm still does not move the way the robot's does, and the camera sits on a human head, not where Memo's is. Did the operators move like the robot, or just do the task the way a human does and leave the model to absorb the difference? Domain gap.

ACT-2

The recipe is large-scale diverse pretraining, then finetuning per task in deployment.

Fold success against pretrain data scale. In-domain sits at 96% with zero pretraining and barely moves. Out-of-domain starts at 14%, is at about 90% with 12% of the data and matches in-domain at 100%. So pretraining buys almost nothing in-domain, and most of the out-of-domain gain comes from the first 12% of the data. (I should try this soon with an SO-101 or something.)

What helped was seeing how they break the task down: object (laundry), task, initial condition. Without a defined initial condition the 99% they report does not mean much.

Two questions I had while watching: large data or small good-quality data, and what subsampling means here (every Nth frame, or picking the good episodes out of the 10M?).

unit economics

They own the whole vertical stack, and the part that matters is the data, which they own and are scaling aggressively. That is the moat, but I am not sure about the rate: it has to be collected at human speed by people wearing gloves, so at some point the bottleneck moves from compute to how many people are wearing gloves this month.

From a conversation I had a couple of weeks ago:

"For the past year we haven't seen a good model come from egocentric data" "UMI and teleoperation (pi-0.5 and sunday / generalist)" "Don't make a bet on ego. (Really low unit economics, swarming the market with this data)"

I wanted numbers on the data cost, so I looked up what the rigs cost.

The glove costs almost exactly what a UMI rig costs, which makes sense given who built both. Against teleop it is roughly 80x cheaper per rig, and you don't need a robot standing next to every operator: you ship a $400 pair to someone's kitchen and the scene diversity comes with it. Ego glasses are in the same price class, but the video has no action labels, so every hour needs pose estimation and retargeting before it is usable.

They run edge infrastructure instead of cloud, because cloud did not work in the home.

Compute on the device is the right call for a home product, but it sets a ceiling. Makes me want to know how big the model actually is.

generalist and pi 0.6

Generalist runs the cheapest capture of the three: 270,000 hours (half a million by the GEN-1 post) from a simpler, non-actuated UMI gripper with a QR code and a camera, no ego-camera at all. Pi 0.6 sticks to teleop, with a Galaxea robot for the mobile case.

So Generalist gets breadth cheaply with a passive gripper, Pi keeps the fidelity of teleop and pays a robot per operator, and Sunday sits in between with an actuated glove.