Teleop Dataset Recording

SO101-Nexus ships with a Gradio-based teleop recorder in the core package. You drive a simulated robot with a physical SO-100 or SO-101 leader arm, and demonstrations are saved as LeRobot v3 datasets ready to push to the Hugging Face Hub.

Quickstart with uvx

If you have uv installed, you can launch the recorder against either backend without setting up a venv. uvx resolves the package and the teleop extra into an ephemeral environment:

# MuJoCo
uvx --from "so101-nexus-mujoco[teleop]" so101-nexus-mujoco teleop \
    --leader-port /dev/ttyACM0

# ManiSkill
uvx --from "so101-nexus-maniskill[teleop]" --prerelease=allow \
    so101-nexus-maniskill teleop --leader-port /dev/ttyACM0

This is the fastest way to try teleop end to end. The rest of this guide assumes you have run a regular install for repeat use, but the uvx command above is enough for a one-off session.

Installation

Teleop deps (lerobot[feetech], gradio, plotly, opencv-python) live in the teleop extra on so101-nexus-core. Install them alongside the backend you want to record against:

# MuJoCo
uv sync --package so101-nexus-mujoco --extra teleop

# ManiSkill
uv sync --package so101-nexus-maniskill --extra teleop

With pip:

# MuJoCo
pip install "so101-nexus-core[teleop]" so101-nexus-mujoco

# ManiSkill
pip install "so101-nexus-core[teleop]" so101-nexus-maniskill

Hardware

• A physical SO-100 or SO-101 leader arm connected via USB.
• A Linux host with access to the leader arm's serial port.

Finding the serial port

Make the device writable if needed, then use the LeRobot port discovery tool:

sudo chmod 666 /dev/ttyACM0
lerobot-find-port

If the recorder starts before the port is writable, the Initialize step now stays open and shows exact recovery commands. Fix the port in another terminal, then click Retry Initialization in the same browser session.

Launching the recorder

Each backend exposes a teleop subcommand:

# MuJoCo
uv run so101-nexus-mujoco teleop --leader-port /dev/ttyACM0

# ManiSkill
uv run so101-nexus-maniskill teleop --leader-port /dev/ttyACM0

A Gradio UI opens in your browser. Most settings live in the UI; the CLI only takes connection-level options.

If initialization fails because /dev/ttyACM* is missing or not writable, teleop now keeps the session alive and prints targeted recovery guidance instead of forcing a browser restart.

CLI arguments

Cameras and recorded fields

The recorder captures both the wrist camera and the overhead camera on every step, so you can decide after the fact which views end up in your dataset without re-recording.

Under Advanced Settings, Dataset fields, a checkbox group selects which fields are persisted:

• observation.state always saved (disabled checkbox)
• action always saved (disabled checkbox)
• task always saved by LeRobot v3 metadata
• observation.images.wrist optional, on by default
• observation.images.overhead optional, on by default

Deselecting an optional field removes it from both the declared dataset feature schema and every recorded frame. observation.state, action, and task stay forced on because LeRobot datasets require them for policy training and task indexing.

Camera resolution

Separate width and height sliders for the wrist and overhead cameras control the recording resolution. Both cameras are wired into the environment's observation config automatically, even when the task's default config only declares one.

Environment customization

Teleop customization uses the same typed config objects as the regular Gymnasium API. The recorder starts from the selected environment's default config, applies UI/profile/factory overrides, then wires the wrist and overhead camera observations required for recording.

The Advanced Settings section exposes common options:

• Apply Environment Customization: enables the UI environment overrides below. Leave it off to use the environment's default config unchanged.
• Pick Object Pool: built-in cube and YCB objects for pick tasks
• Pick Distractors: number of distractors to place in pick scenes
• Ground Colors and Robot Colors: sampled per reset when multiple colors are selected
• Spawn Min Radius, Spawn Max Radius, and Spawn Angle Half Range (deg): spawn-region controls shared by supported tasks
• Reset Settle Frames: no-op frames advanced after reset before the first teleop observation and recorded frame. The default is 5.
• Pick-and-Place Cube Colors and Pick-and-Place Target Colors: sampled cube/target colors for pick-and-place tasks

Options that do not apply to the selected environment are ignored after customization is enabled. For example, pick-object settings apply to PickConfig environments, while pick-and-place color settings apply to PickAndPlaceConfig environments.

Reset Settle Frames is applied to the environment config before camera observations are wired, so the first saved demonstration frame is captured after reset settling.

Config profiles

Use a profile when you want repeatable recording setups. Profiles may be JSON or TOML and are merged in this order:

1. flat top-level override keys
2. common
3. matching task section, either pick or pick_and_place
4. envs.<env_id>

Later sections override earlier sections.

[common]
ground_colors = ["gray", "white"]
robot_colors = ["yellow"]
spawn_min_radius = 0.15
spawn_max_radius = 0.25
reset_settle_frames = 5

[pick]
n_distractors = 1
objects = [
  { type = "cube", color = "green" },
  { type = "ycb", model_id = "011_banana" },
  { type = "ycb", model_id = "058_golf_ball" },
]

[pick_and_place]
cube_colors = ["red", "green"]
target_colors = ["blue"]

[envs.MuJoCoPickLift-v1]
spawn_angle_half_range_deg = 60.0

Launch with:

uv run so101-nexus-mujoco teleop \
    --leader-port /dev/ttyACM0 \
    --env-config-profile teleop-profile.toml

The profile object schema matches the object classes documented in Scene Objects. String specs such as cube:blue and ycb:011_banana are supported for built-in UI/CLI object choices. Mesh objects use mapping syntax because file paths may contain separator characters:

[[pick.objects]]
type = "mesh"
name = "custom widget"
collision_mesh_path = "/path/to/collision.stl"
visual_mesh_path = "/path/to/visual.obj"
mass = 0.02
scale = 1.0

MeshObject is MuJoCo-only.

Config factories

For advanced setups, provide a Python factory:

uv run so101-nexus-mujoco teleop \
    --leader-port /dev/ttyACM0 \
    --env-config-factory my_project.teleop_configs:build_config

The factory receives (env_id, base_config). It may return a config object:

from so101_nexus_core import CubeObject, PickConfig, YCBObject


def build_config(env_id, base_config):
    return PickConfig(
        objects=[CubeObject(color="green"), YCBObject("011_banana")],
        n_distractors=1,
    )

It may also return a dict of gym.make kwargs. If the dict includes "config", teleop still wires the recording cameras after the factory runs.

Custom environments

Custom environments must be registered with Gymnasium before teleop builds the dropdown. Import the registration module and list the environment ID:

uv run so101-nexus-mujoco teleop \
    --leader-port /dev/ttyACM0 \
    --env-module my_project.envs \
    --extra-env-id CustomPick-v1

Custom environments should accept SO-100/SO-101 six-joint actions in the standard joint order, expose current qpos through _get_current_qpos() or agent.robot.get_qpos(), and return camera observations compatible with the selected recording fields. If the unwrapped environment exposes task_description, teleop records it as the required LeRobot task field.

Session flow

1. Configure. Pick the env, robot type, episode count, FPS, camera resolutions, action space, countdown, and dataset fields.
2. Initialize. The app connects to the leader arm and creates the dataset. If the port is unavailable or blocked by permissions, fix it and retry from the same page.
3. Record. A countdown plays, then the simulation mirrors the leader arm. The live wrist feed renders in the UI.
4. Review. After each episode, the UI shows a video replay and a joint-state plot.
5. Approve or discard. Accept to write the episode into the dataset, or discard and re-record.
6. Push. When all episodes are done, push to the Hugging Face Hub from the UI.

Recorded dataset format

Datasets use the LeRobot v3 SO follower schema. Depending on your selection, frames include:

• observation.state: follower readback (6D), always present
• action: commanded absolute joint positions (6D) by default, always present
• observation.images.wrist: wrist camera frames, optional
• observation.images.overhead: overhead camera frames, optional
• task: language description, always present

Body joints are stored in LeRobot degree units and the gripper is stored in RANGE_0_100 percent, matching sim_so_follower and SO100/SO101 MolmoAct2 normalization. observation.state comes from simulated follower readback, not the leader echo.

On first recording, the app creates a simulator-only calibration file at $HF_LEROBOT_CALIBRATION/robots/sim_so_follower/teleop_sim.json (or the LeRobot default calibration root when HF_LEROBOT_CALIBRATION is unset). No physical follower calibration is required for simulator teleop.

Troubleshooting

For Ubuntu viewer/runtime issues such as libdecor-gtk warnings, see Teleoperation Troubleshooting.