Robot Learning on a Bi-manual Setup

Over the past six months, the Robot Learning team at the ETH Robotics Club devoted its efforts to building a robust, low-cost bi-manual arm setup, collecting high-quality trajectory data, and training imitation learning models to perform complex, short-horizon manipulation tasks autonomously.

Phase 1: Iterating on Hardware and Teleoperation

Our journey began with a pair of AgileX's lightweight 6DoF PiPER arms. For teleoperation, we initially deployed a VR framework using a Meta Quest 3 headset, paired with wrist-mounted Innomaker (U20CAM-1080P-S1) cameras.

However, we quickly hit a bottleneck, as our setup lacked the precision and structural robustness required for reliable data collection. After noticing this during data collection we pivoted our hardware setup, swapping the PiPER arms for I2RT's YAM arms. We further engineered our own passive leader arms for bilateral teleoperation, using a series of Dynamixel motors. Lastly, we built a rigid custom aluminum extrusion rig to support and keep the YAM arms and camera fixed in place. With the new and adjusted hardware setup, we observed more reliable and robust results during teleoperation.

If an end-effector drives into the table, the lack of compliance can easily destroy the gripper, or even more critically cause damage to the motors of the YAM arms. Thus, a first necessary modification was to replace the standard rigid end-effector grippers with a pair of Soft Robotics Fin Ray grippers printed out of TPU. Integrating these soft grippers came with the added benefit of improved grasping capabilities (as TPU has a high coefficient of friction compared to PLA) for irregular object geometries, while simultaneously satisfying our compliance and safety requirements. The flexible structure naturally conforms to objects for a more secure grasp while introducing a mechanical safety buffer that absorbs vibrations, shocks and impacts. We designed these grippers to be completely hotswappable onto our end-effectors, to allow for rapid future experimentation and reduce downtime.

Following the new year, we shifted our focus toward system maturity, ergonomics, and tackling more complex task horizons. We overhauled the passive leader arms, swapping out the weak motors for significantly stronger Dynamixel variants with higher stall torques. This upgrade was important for us, as it enabled active gravity compensation, drastically reducing operator fatigue and allowing for much smoother, more natural teleoperation.

Phase 2: First Validation & Towel Test

With the hardware stabilized, we set out to validate the entire pipeline (from data collection to policy deployment) before the end of the semester. We collected a dataset of 250 trajectories of a short-horizon task: towel folding. Using this data, we trained an Action Chunking Transformer (ACT) model. During autonomous inference testing, the model achieved an impressive 85% success rate, successfully validating our hardware rigidity, camera calibration, and the software pipeline.

The data foundation the team trains on. Raw YAMS teleoperation episodes → quality validation → standard format → train/val/test splits. After 300+ episodes I learned it the hard way: more demos isn't more signal — a wrist camera slipping, the top-down angle drifting between sessions, a calibration nudged mid-batch all silently poison the dataset. So validation now gates the pipeline. 863 demos curated across 9 datasets (347 towel, 516 carton-box), inspected through a LeRobot visualizer, with the same camera-drift checks keeping every batch consistent.

12-panel Plotly dashboard for diagnosing what the model sees before you trust a checkpoint: ACT readiness score, chunk disagreement heatmap, episode×episode DTW similarity, plus a camera-drift analysis surfacing 44 worst-pair comparisons across the dataset.

With an upgraded system, we could now collect trajectories for a highly dexterous, multi-stage, long-horizon task: picking up a handheld fan, placing it neatly inside a box, and closing the box lid. Because this task demanded millimeter-level precision and a sequence of distinct manipulation phases, it presented a much steeper challenge for the model than towel folding, serving as an excellent benchmark for our ongoing software iterations. Due to the dexterity of this task, its success rate was not as high as for the towel folding task.

Two ACT policies running on YAMS

What are we working on now / where are we headed

• Orca Hands
• Daimon Robotics Fingertip Sensor: Force / Torque Feedback
• ETHRC Centaur

Moreover, our current efforts lie in developing our own framework based on Video Action Models and in building a mobile robot version of the bimanual 6DoF setup. Stay tuned for the ETHRC Centaur!