Robot Learning on a Bi-manual Setup

Over the past six months, the Robot Learning team at the ETH Robotics Club built a low-cost bimanual arm setup, collected trajectory data with it and trained imitation learning models on short-horizon manipulation tasks.

Phase 1: iterating on hardware and teleoperation

We started with a pair of AgileX's lightweight 6DoF PiPER arms. For teleoperation we first used a VR setup with a Meta Quest 3 headset and wrist-mounted Innomaker (U20CAM-1080P-S1) cameras.

The setup was not precise or rigid enough for reliable data collection, so we swapped the PiPER arms for I2RT's YAM arms. We also built our own passive leader arms for bilateral teleoperation out of Dynamixel motors, and a rigid aluminum extrusion rig that keeps the YAM arms and the camera fixed in place. Teleoperation got noticeably more reliable with the new hardware.

If an end-effector drives into the table, a rigid gripper breaks, or worse, the YAM motors take the damage. So the first modification was replacing the standard rigid grippers with Fin Ray grippers printed in TPU. They also grip better: TPU has a higher coefficient of friction than PLA, and the fins conform to irregular objects. We designed them to be hot-swappable so we can try other grippers without much downtime.

After the new year we rebuilt the leader arms with stronger Dynamixel motors. The higher stall torque made active gravity compensation possible, which makes teleoperation much less tiring and a lot smoother.

Phase 2: first validation, the towel test

With the hardware stable, we wanted to validate the whole pipeline, from data collection to policy deployment, before the end of the semester. We collected 250 trajectories of towel folding and trained an Action Chunking Transformer (ACT) on them. Running autonomously, the policy folded the towel in 85% of attempts, which was enough to trust the rig, the camera calibration and the software pipeline.

The data pipeline: raw YAMS teleoperation episodes → quality validation → standard format → train/val/test splits. After 300+ episodes I learned that more demos do not automatically mean better data. A slipped wrist camera, a top-down angle that drifted between sessions or a calibration nudged mid-batch does not show up until training, so every batch now goes through validation first. We have 863 demos across 9 datasets (347 towel, 516 carton box), inspected in the LeRobot visualizer with the same camera-drift checks on every batch.

A 12-panel Plotly dashboard for checking what the model sees before trusting a checkpoint: an ACT readiness score, a chunk disagreement heatmap, episode-by-episode DTW similarity and a camera-drift analysis that shows the 44 worst pairs in the dataset.

With the upgraded system we collected trajectories for a longer task: pick up a handheld fan, place it inside a box and close the lid. It needs millimeter-level precision and several distinct phases, so it is much harder for the model than towel folding, and the success rate was lower. We use it as the benchmark for our software iterations.

Two ACT policies running on YAMS

What are we working on now / where are we headed

• Orca Hands
• Daimon Robotics Fingertip Sensor: Force / Torque Feedback
• ETHRC Centaur

We are also developing our own framework based on Video Action Models and building a mobile version of the bimanual setup, the ETHRC Centaur.