robotic-lab

Simulated SO-101 robot-arm mini-lab: keyboard teleoperation → episode recording → LeRobot v2.1 dataset → ACT (Action Chunking Transformer) policy training.

Task: pick the red cube, drop it in the blue bucket — in MuJoCo, schema-identical to the pretrained checkpoint davidlinjiahao/lerobot_so101_base_sim_pickplace + its source dataset lerobot_batch_001, so your collected demos can warm-start it directly.

What's included

Path	Purpose
so101_task/teleop_pick.py	Keyboard teleop + 30 Hz recorder (own GLFW window — no macOS Accessibility permission needed; pynput is avoided: it crashes on macOS 26)
so101_task/scene_task.xml	MuJoCo scene: SO-101 arm, table, cube, bucket, tuned front camera (480×640)
so101_task/to_lerobot.py	npz → LeRobot v2.1 dataset (parquet + h264 mp4 + meta), timestamp-synchronized
so101_task/pick_lib.py	IK/CEM/shared sim helpers
so101_task/record_selftest.py	Headless recorder sanity test
so101_task/probe1..13.py, task_search.py	Diagnostics: why scripted IK grip fails on this arm (why value-based clamping)
models/SO-ARM100	Submodule → TheRobotStudio/SO-ARM100 (MJCF/URDF source of truth)

Episode format (episodes/episode_XXXX.npz, 30 Hz):

• observation.state (12): 6 joints + cube position (3) + cube-to-bin vector (3)
• action (6), observation.images.front 480×640 RGB
• timestamps derived as frame_index / fps (lerobot's video-alignment rule)

Setup

git clone --recursive https://github.com/aruninf/robotic-lab.git
cd robotic-lab
python3.12 -m venv .venv
.venv/bin/pip install "mujoco==3.10.*" numpy pyarrow "imageio[ffmpeg]" pynput

MuJoCo is pinned to 3.10 — newer releases dropped Intel-Mac wheels. Tested on Intel macOS.

Usage

1. Collect demos (you teleoperate — keyboard only, no webcam):

cd so101_task && ../.venv/bin/python teleop_pick.py

Keys	Action
w/s a/d q/e z/x c/v	shoulder-lift / pan / elbow / wrist-flex / wrist-roll
hold SPACE	close gripper (release to open)
t	start / stop recording a take
r	reset scene
ESC	quit (auto-saves an in-progress take)

Click the window once if keys don't register. Recipe per take: hover over cube → descend onto it → hold SPACE ~2 s → lift → carry over bucket → release → t.

2. Convert to a LeRobot dataset:

../.venv/bin/python to_lerobot.py

Outputs lerobot_sod101_pick/ with the exact v2.1 layout of lerobot_batch_001 — loadable by lerobot 0.6.x.

3. Train (fine-tune the ACT): needs torch ≥ 2.7 with CUDA — run in Linux Docker on CPU ×small budget or on a cloud GPU (Colab T4 / RunPod). Warm-start from davidlinjiahao/lerobot_so101_base_sim_pickplace.

Status / roadmap

• Teleop + recorder verified end-to-end (real episodes recorded & validated) Click the window once if keys don't register. Recipe per take: hover over cube → descend onto it → hold SPACE ~2 s → lift → carry over bucket → release → t.

Input devices: keyboard vs alternatives

Keyboard teleop works, but binary keys are the hardest way to drive precise motion. Alternatives:

Device	Cost	Notes
Gamepad (Xbox / PS / 8BitDo)	~$0–25 (if you own one)	Analog sticks = smooth proportional steering; planned: pygame branch in teleop_pick.py
3Dconnexion SpaceMouse	~$80–130	3-D push/puck control, built-in lerobot teleop support
Real SO-101 Leader arm	~$130–180	The proper way: grab its joints, the simulated arm mirrors you incl. gripper — how the reference lerobot demos were collected. Reusable later to drive a real follower arm (sim-to-real)

Webcam hand-gesture teleop (GitHub refs)

Gesture control = MediaPipe hand landmarks (~30 lines, CPU webcam) + mapping to joint targets. Existing projects are wired to their own hardware stacks, not directly to this sim:

Reference	Approach	Glue needed for this repo
mernaahany/teleoperation-robot-arm	MediaPipe hand pos + pinch = gripper → arm joints	reuse mapping logic → now built in: python teleop_pick.py --gestures
AFK-Kireeti/Lerobot-SO101	Hand tracking + ArUco cube + IMU, ROS2/MoveIt	Heavy: needs Raspberry Pi 5 + ROS2 stack
Eventual-Inc/daft-physical-ai hands demo	MediaPipe over LeRobot datasets (analysis, not control)	Wrong direction — detects hands in recorded data
Hand-shadowing paper (arXiv 2603.11383)	RealSense depth cam on glasses → IK → SO-ARM101, PyBullet preview	Best fit technically; needs ~$250 Intel RealSense

Planned for this repo: a MediaPipe gesture branch done: --gestures flag — open hand steers via warm-seeded position-only IK (pick_lib), pinch (thumb–index) grips, hysteresis + low-pass smoothing, no-hand frames freeze controls, keyboard stays live as fallback. Caveat: single-webcam depth is a palm-size proxy and tracking jitters — for clean training data, leader arm or gamepad still beats gestures.

<tool_call>edit<arg_key>newString</arg_key><arg_value>- [x] Converter byte-accurate vs reference dataset

• ≥30 quality episodes (currently a handful)
• gamepad teleop branch (cheap analog input for easier demo collection)
• webcam gesture teleop branch (MediaPipe hand tracking → sim joints)
• ACT fine-tune on GPU, evaluate in-sim
• (stretch) visual policy input — the front camera feed is already recorded for it

License: code MIT; models/SO-ARM100 keeps its upstream license.