Teleoperation

SO101-Nexus ships a Gradio teleop recorder. A physical SO-100 or SO-101 leader arm drives a simulated follower, and every approved episode is written to a LeRobot v3 dataset you can push to the Hugging Face Hub. That dataset is the artifact the clone and reinforce stages consume.

Quickstart

uvx --from "so101-nexus[teleop]" so101-nexus teleop \
    --leader-port /dev/ttyACM0

A Gradio UI opens in your browser. Most settings live in the UI; the CLI only takes connection-level options. For a repeat-use install of the teleop extra, see Installation.

Hardware and the serial port

You need a physical SO-100 or SO-101 leader arm on USB and a host with access to its serial port. If the device is not writable, fix permissions and confirm the port:

sudo chmod 666 /dev/ttyACM0
lerobot-find-port

The Configure step reports leader-port reachability before you fill in the rest of the form, with a Recheck Port button, and blocks Initialize Session while the port is not ready. A connect-time failure keeps the session alive on the Initialize step with recovery guidance, so you can fix the port in another terminal and click Retry Initialization.

Session flow

1. Configure. Pick the environment, robot type, episode count, FPS, camera resolutions, action space, countdown, dataset fields, Max Steps, and Success Hold.
2. Initialize. The app connects to the leader arm and creates the dataset.
3. Record. A countdown plays, then the simulation mirrors the leader arm. The live wrist feed renders in the UI.
4. Review. After each episode, the UI shows a video replay and a joint-state plot.
5. Approve or discard. Accept to write the episode into the dataset, or re-record.
6. Push. When all episodes are done, push to the Hugging Face Hub from the UI.

CLI arguments

Cameras and recorded fields

Both the wrist and overhead cameras are captured on every step, so you can decide after the fact which views end up in your dataset. Both are wired into the environment's observation config automatically even when the task's default config declares only one, and separate width and height sliders set each camera's resolution.

Under Advanced Settings, Dataset fields, a checkbox group selects which fields are persisted; deselecting an optional field drops it from the declared feature schema and from every recorded frame. observation.state, action, reward, success, done, and task stay forced on because LeRobot datasets require them for policy training and task indexing.

Recorded dataset format

Datasets use the LeRobot v3 SO follower schema. Always present:

• observation.state: follower readback, 6D
• action: commanded absolute joint positions, 6D
• reward: scalar transition reward from the same environment step as action
• success and done: episode success and termination flags
• task: language description, from LeRobot v3 metadata

Optional, on by default: observation.images.wrist, observation.images.overhead, and observation.environment_state (privileged low-dim state: TCP and object pose, grasp, offsets).

Body joints are stored in LeRobot degree units and the gripper in RANGE_0_100 percent, matching sim_so_follower and SO100/SO101 MolmoAct2 normalization. observation.state is simulated follower readback, not the leader echo.

On finalize, before any Hub push, the app writes the environment id and customization overrides to meta/so101_nexus_env.json inside the dataset. That file is itself a valid config profile, so --env-config-profile meta/so101_nexus_env.json reproduces a recording's environment. It captures UI and CLI overrides only; a separate --env-config-profile used at record time is not merged in.

On first recording the app creates a simulator-only calibration file at $HF_LEROBOT_CALIBRATION/robots/sim_so_follower/teleop_sim.json, or under the LeRobot default calibration root when that variable is unset. No physical follower calibration is needed.

Environment customization

Teleop uses the same typed config objects as the regular Gymnasium API. The recorder starts from the selected environment's default config, applies UI, profile, and factory overrides, then wires the camera observations required for recording.

Advanced Settings exposes the common options. Turn on Apply Environment Customization to enable them; leave it off to use the default config unchanged.

Options that do not apply to the selected environment are ignored: pick-object settings apply to PickConfig, pick-and-place colors to PickAndPlaceConfig, stack-cube colors to StackCubeConfig. Field names and defaults are in Config reference.

Config profiles

Use a profile for a repeatable recording setup. Profiles may be JSON or TOML and are merged in this order, with later sections overriding earlier ones: flat top-level override keys, common, the matching task section (pick, pick_and_place, or stack_cube), then envs.<env_id>.

[common]
ground_colors = ["gray", "white"]
robot_colors = ["yellow"]
spawn_min_radius = 0.15
spawn_max_radius = 0.25
reset_settle_frames = 5

[pick]
n_distractors = 1
objects = [
  { type = "cube", color = "green" },
  { type = "ycb", model_id = "011_banana" },
  { type = "ycb", model_id = "058_golf_ball" },
]

[pick_and_place]
cube_colors = ["red", "green"]
target_colors = ["blue"]

[stack_cube]
cube_a_colors = ["red", "orange"]
cube_b_colors = ["blue", "green"]

[envs.MuJoCoPickLift-v1]
spawn_angle_half_range_deg = 60.0

uv run so101-nexus teleop \
    --leader-port /dev/ttyACM0 \
    --env-config-profile teleop-profile.toml

The profile object schema matches the classes in Scene Objects. String specs such as cube:blue and ycb:011_banana work for built-in object choices. Mesh objects use mapping syntax because file paths may contain separator characters:

[[pick.objects]]
type = "mesh"
name = "custom widget"
collision_mesh_path = "/path/to/collision.stl"
visual_mesh_path = "/path/to/visual.obj"
mass = 0.02
scale = 1.0

Config factories

For advanced setups, pass a Python factory with --env-config-factory my_project.teleop_configs:build_config. It receives (env_id, base_config) and may return a config object:

from so101_nexus import CubeObject, PickConfig, YCBObject


def build_config(env_id, base_config):
    return PickConfig(
        objects=[CubeObject(color="green"), YCBObject("011_banana")],
        n_distractors=1,
    )

It may also return a dict of gym.make kwargs. If the dict includes "config", teleop still wires the recording cameras after the factory runs.

Custom environments

Custom environments must be registered with Gymnasium before teleop builds the dropdown. Import the registration module and list the environment ID:

uv run so101-nexus teleop \
    --leader-port /dev/ttyACM0 \
    --env-module my_project.envs \
    --extra-env-id CustomPick-v1

A custom environment should accept SO-100/SO-101 six-joint actions in the standard joint order, expose current qpos through _get_current_qpos(), and return camera observations compatible with the selected recording fields. If the unwrapped environment exposes task_description, teleop records it as the required LeRobot task field.

Published demonstration datasets

Troubleshooting