Teleop Dataset Recording

SO101-Nexus ships with a Gradio-based teleop recorder. You drive a simulated robot with a physical SO-100 or SO-101 leader arm, and demonstrations are saved as LeRobot v3 datasets ready to push to the Hugging Face Hub.

Quickstart with uvx

If you have uv installed, you can launch the recorder without setting up a venv. uvx resolves the package and the teleop extra into an ephemeral environment:

uvx --from "so101-nexus[teleop]" so101-nexus teleop \
    --leader-port /dev/ttyACM0

This is the fastest way to try teleop end to end. The rest of this guide assumes you have run a regular install for repeat use, but the uvx command above is enough for a one-off session.

Installation

Teleop deps (lerobot[feetech], gradio, plotly, opencv-python) live in the teleop extra on so101-nexus. Install them with:

uv sync --extra teleop

With pip:

pip install "so101-nexus[teleop]"

Hardware

• A physical SO-100 or SO-101 leader arm connected via USB.
• A Linux host with access to the leader arm's serial port.

Finding the serial port

Make the device writable if needed, then use the LeRobot port discovery tool:

sudo chmod 666 /dev/ttyACM0
lerobot-find-port

The Configure step shows a leader-port status banner up front: it reports whether the device exists and is readable and writable before you fill in the rest of the form. Click Recheck Port after fixing permissions to refresh it, and Initialize Session is blocked with the same guidance while the port is not ready. If the recorder still cannot open the arm at connect time, the Initialize step stays open and shows the connection error, with exact recovery commands when the port is missing or blocked by permissions. Fix the port in another terminal, then click Retry Initialization in the same browser session.

Launching the recorder

The teleop subcommand launches the recorder:

uv run so101-nexus teleop --leader-port /dev/ttyACM0

A Gradio UI opens in your browser. Most settings live in the UI; the CLI only takes connection-level options.

If initialization fails because /dev/ttyACM* is missing or not writable, teleop now keeps the session alive and prints targeted recovery guidance instead of forcing a browser restart.

CLI arguments

Cameras and recorded fields

The recorder captures both the wrist camera and the overhead camera on every step, so you can decide after the fact which views end up in your dataset without re-recording.

Under Advanced Settings, Dataset fields, a checkbox group selects which fields are persisted:

• observation.state always saved (disabled checkbox)
• action always saved (disabled checkbox)
• task always saved by LeRobot v3 metadata
• reward always saved from the same environment step as the frame's action
• success always saved (episode success flag)
• done always saved (episode termination flag)
• observation.environment_state optional, on by default (privileged low-dim state: TCP/object pose, grasp, and offsets)
• observation.images.wrist optional, on by default
• observation.images.overhead optional, on by default

Deselecting an optional field removes it from both the declared dataset feature schema and every recorded frame. observation.state, action, reward, success, done, and task stay forced on because LeRobot datasets require them for policy training and task indexing.

Camera resolution

Separate width and height sliders for the wrist and overhead cameras control the recording resolution. Both cameras are wired into the environment's observation config automatically, even when the task's default config only declares one.

Environment customization

Teleop customization uses the same typed config objects as the regular Gymnasium API. The recorder starts from the selected environment's default config, applies UI/profile/factory overrides, then wires the wrist and overhead camera observations required for recording.

The Advanced Settings section exposes common options:

• Apply Environment Customization: enables the UI environment overrides below. Leave it off to use the environment's default config unchanged.
• Pick Object Pool: built-in cube and YCB objects for pick tasks
• Pick Distractors: number of distractors to place in pick scenes
• Ground Colors and Robot Colors: sampled per reset when multiple colors are selected
• Spawn Min Radius, Spawn Max Radius, and Spawn Angle Half Range (deg): spawn-region controls shared by supported tasks
• Reset Settle Frames: no-op frames advanced after reset before the first teleop observation and recorded frame. The default is 5.
• Pick-and-Place Cube Colors and Pick-and-Place Target Colors: sampled cube/target colors for pick-and-place tasks
• Max Steps: overrides the episode's step cap (default 1024)
• Success Hold (s): seconds to keep recording after the env reports success before auto-stopping (default 0.5 s, 0 = stop immediately)

Options that do not apply to the selected environment are ignored after customization is enabled. For example, pick-object settings apply to PickConfig environments, while pick-and-place color settings apply to PickAndPlaceConfig environments.

Reset Settle Frames is applied to the environment config before camera observations are wired, so the first saved demonstration frame is captured after reset settling.

Config profiles

Use a profile when you want repeatable recording setups. Profiles may be JSON or TOML and are merged in this order:

1. flat top-level override keys
2. common
3. matching task section, either pick or pick_and_place
4. envs.<env_id>

Later sections override earlier sections.

[common]
ground_colors = ["gray", "white"]
robot_colors = ["yellow"]
spawn_min_radius = 0.15
spawn_max_radius = 0.25
reset_settle_frames = 5

[pick]
n_distractors = 1
objects = [
  { type = "cube", color = "green" },
  { type = "ycb", model_id = "011_banana" },
  { type = "ycb", model_id = "058_golf_ball" },
]

[pick_and_place]
cube_colors = ["red", "green"]
target_colors = ["blue"]

[envs.MuJoCoPickLift-v1]
spawn_angle_half_range_deg = 60.0

Launch with:

uv run so101-nexus teleop \
    --leader-port /dev/ttyACM0 \
    --env-config-profile teleop-profile.toml

The profile object schema matches the object classes documented in Scene Objects. String specs such as cube:blue and ycb:011_banana are supported for built-in UI/CLI object choices. Mesh objects use mapping syntax because file paths may contain separator characters:

[[pick.objects]]
type = "mesh"
name = "custom widget"
collision_mesh_path = "/path/to/collision.stl"
visual_mesh_path = "/path/to/visual.obj"
mass = 0.02
scale = 1.0

MeshObject is MuJoCo-only.

Config factories

For advanced setups, provide a Python factory:

uv run so101-nexus teleop \
    --leader-port /dev/ttyACM0 \
    --env-config-factory my_project.teleop_configs:build_config

The factory receives (env_id, base_config). It may return a config object:

from so101_nexus import CubeObject, PickConfig, YCBObject


def build_config(env_id, base_config):
    return PickConfig(
        objects=[CubeObject(color="green"), YCBObject("011_banana")],
        n_distractors=1,
    )

It may also return a dict of gym.make kwargs. If the dict includes "config", teleop still wires the recording cameras after the factory runs.

Custom environments

Custom environments must be registered with Gymnasium before teleop builds the dropdown. Import the registration module and list the environment ID:

uv run so101-nexus teleop \
    --leader-port /dev/ttyACM0 \
    --env-module my_project.envs \
    --extra-env-id CustomPick-v1

Custom environments should accept SO-100/SO-101 six-joint actions in the standard joint order, expose current qpos through _get_current_qpos(), and return camera observations compatible with the selected recording fields. If the unwrapped environment exposes task_description, teleop records it as the required LeRobot task field.

Session flow

1. Configure. Pick the env, robot type, episode count, FPS, camera resolutions, action space, countdown, dataset fields, Max Steps, and Success Hold. A leader-port status banner reports up front whether the configured serial port is reachable, with a Recheck Port button to refresh after fixing it.
2. Initialize. The app connects to the leader arm and creates the dataset. If the port is unavailable or blocked by permissions, fix it and retry from the same page.
3. Record. A countdown plays, then the simulation mirrors the leader arm. The live wrist feed renders in the UI.
4. Review. After each episode, the UI shows a video replay and a joint-state plot.
5. Approve or discard. Accept to write the episode into the dataset, or discard and re-record.
6. Push. When all episodes are done, push to the Hugging Face Hub from the UI.

Recorded dataset format

Datasets use the LeRobot v3 SO follower schema. Depending on your selection, frames include:

• observation.state: follower readback (6D), always present
• action: commanded absolute joint positions (6D) by default, always present
• reward: scalar transition reward from the same environment step as action, always present
• success: episode success flag, always present
• done: episode termination flag, always present
• observation.images.wrist: wrist camera frames, optional
• observation.images.overhead: overhead camera frames, optional
• observation.environment_state: privileged low-dim state (TCP/object pose, grasp, and offsets), optional
• task: language description, always present

Body joints are stored in LeRobot degree units and the gripper is stored in RANGE_0_100 percent, matching sim_so_follower and SO100/SO101 MolmoAct2 normalization. observation.state comes from simulated follower readback, not the leader echo.

On finalize (and before any Hub push), the app writes the environment id and customization overrides to meta/so101_nexus_env.json inside the dataset. The file is itself a valid config profile, so you can reproduce a recording's environment by passing it back as --env-config-profile meta/so101_nexus_env.json. It captures UI/CLI overrides; contributions from a separate --env-config-profile used at record time are not merged in.

On first recording, the app creates a simulator-only calibration file at $HF_LEROBOT_CALIBRATION/robots/sim_so_follower/teleop_sim.json (or the LeRobot default calibration root when HF_LEROBOT_CALIBRATION is unset). No physical follower calibration is required for simulator teleop.

Example rollout datasets

Recorded MuJoCo teleoperation rollouts are available on Hugging Face:

Troubleshooting

For Ubuntu viewer/runtime issues such as libdecor-gtk warnings, see Teleoperation Troubleshooting.