LeRobot compatibility

SO101-Nexus treats the LeRobot ecosystem as a first-class peer. When LeRobot ships an abstraction, we use it directly rather than reimplementing one of our own.

What this means in practice

• Processor pipelines. Teleop conversions and optional env-observation transforms are implemented as lerobot.processor.DataProcessorPipeline instances composed of steps that subclass ActionProcessorStep, ObservationProcessorStep, and friends. Every custom step is registered with ProcessorStepRegistry, which means pipelines are serializable via save_pretrained and shareable on the Hugging Face Hub.
• Observation conventions. When you opt into the wrapper, observations follow LeRobot's canonical keys: observation.state for the proprioceptive state vector and observation.images.<name> for camera frames in CHW float32 form.
• Hardware drivers. SO leader and follower configs come straight from lerobot.teleoperators.so_leader and lerobot.robots.so_follower. We do not wrap or re-export them.
• Datasets. Recordings use LeRobotDataset directly. The Gradio teleop recorder and the LeRobot CLI adapter both use the simulated SO follower conventions for action/state units and camera keys.

Recording with lerobot-record

Install SO101-Nexus with the teleop extra so lerobot[feetech] is available, then load the adapter explicitly with LeRobot's plugin import hook:

lerobot-record \
  --robot.discover_packages_path=so101_nexus.lerobot_adapter \
  --robot.type=sim_so_follower \
  --robot.env_id=MuJoCoTouch-v1 \
  --robot.id=my_robot \
  --robot.calibration_dir=~/.cache/huggingface/lerobot/calibration/robots/so_follower \
  --robot.use_degrees=true \
  --teleop.type=so101_leader \
  --teleop.port=/dev/ttyACM0 \
  --teleop.id=my_leader \
  --dataset.repo_id=user/my_sim_reach \
  --dataset.num_episodes=10 \
  --dataset.single_task="reach the target"

--robot.discover_packages_path is a parser hook, not a RobotConfig field. It imports so101_nexus.lerobot_adapter before config parsing so the LeRobot @register_subclass decorators for sim_so_follower and sim cameras run.

The default is --robot.use_degrees=true, matching LeRobot 0.5 SO follower and SO leader defaults. Percent mode is also supported when both sides opt in:

--robot.use_degrees=false --teleop.use_degrees=false

Keep --robot.use_degrees=true for compatibility with allenai/MolmoAct2-SO100_101. That checkpoint expects action and observation.state to be six-element absolute joint-pose vectors with body joints in LeRobot degree units and the gripper in RANGE_0_100; --robot.use_degrees=false records a valid LeRobot dataset, but in a different body-joint unit space.

Use a real SO follower calibration when you want the same normalized action/state semantics as a physical follower. Record that once with upstream LeRobot, then point --robot.calibration_dir at the directory containing the follower JSON:

lerobot-calibrate \
  --robot.type=so101_follower \
  --robot.port=/dev/ttyACM1 \
  --robot.id=my_robot

The physical leader has its own calibration path. If it is not calibrated yet, calibrate it separately before recording:

lerobot-calibrate \
  --teleop.type=so101_leader \
  --teleop.port=/dev/ttyACM0 \
  --teleop.id=my_leader

For simulator-only tests, create an explicit synthetic follower calibration file:

from pathlib import Path
from so101_nexus.lerobot_adapter.synthetic_calibration import write_synthetic_calibration

write_synthetic_calibration(Path("calibration"), "my_robot")

Synthetic calibration files use LeRobot's normal MotorCalibration schema and the SO101 motor ids 1..6, but they are not a replacement for measured physical calibration data.

The adapter is validated against the MuJoCo backend, reading simulator joint positions and writing actions through it.

Recording with the Gradio teleop app

The Gradio recorder writes the same core schema by default: action and observation.state are six-element absolute joint vectors, body joints are in degrees, and the gripper is in RANGE_0_100. Camera features are stored as observation.images.wrist and observation.images.overhead.

The recorder drives the simulator through SimSOFollower.send_action() and reads state through SimSOFollower.get_observation(), so observation.state is follower readback rather than the leader command echo. It creates a simulator-only calibration file automatically at $HF_LEROBOT_CALIBRATION/robots/sim_so_follower/teleop_sim.json when missing.

Using LeRobot processors with SO101-Nexus envs

LeRobotEnvWrapper requires a Dict observation space. Configure the env with at least one camera component (or pick another component beyond the default state-only set) before wrapping it.

import so101_nexus.mujoco  # noqa: F401
from so101_nexus import JointPositions, PickConfig, WristCamera
from so101_nexus.processors import make_lerobot_env

config = PickConfig(
    obs_mode="visual",
    observations=[JointPositions(), WristCamera(width=224, height=224)],
)
env = make_lerobot_env("MuJoCoPickLift-v1", config=config, render_mode="rgb_array")
obs, _ = env.reset()
# obs is now {"observation.state": ..., "observation.images.wrist": ..., ...}

To customize the pipeline, build one yourself and pass it in:

import gymnasium as gym
from lerobot.processor import DataProcessorPipeline, RenameObservationsProcessorStep
from so101_nexus import JointPositions, PickConfig, WristCamera
from so101_nexus.processors import Hwc2ChwImageObservationStep, LeRobotEnvWrapper

config = PickConfig(
    obs_mode="visual",
    observations=[JointPositions(), WristCamera(width=224, height=224)],
)
pipeline = DataProcessorPipeline(
    steps=[
        RenameObservationsProcessorStep(
            rename_map={
                "state": "observation.state",
                "wrist_camera": "observation.images.wrist",
            }
        ),
        Hwc2ChwImageObservationStep(image_keys=("observation.images.wrist",)),
    ]
)
env = LeRobotEnvWrapper(gym.make("MuJoCoPickLift-v1", config=config), pipeline=pipeline)

Custom processor pipelines

The default leader pipeline (make_default_leader_action_pipeline) is still available for callers that need the historical conversion path: degrees from the leader, radians on the way to a raw simulator env, with a wrist-roll calibration shift. The Gradio recorder no longer uses that pipeline for dataset recording; it routes through SimSOFollower so stored actions and states stay in LeRobot SO follower units.