Teleoperation

SO101-Nexus ships a Gradio teleop recorder. A physical SO-100 or SO-101 leader arm drives a simulated follower, and every approved episode is written to a LeRobot v3 dataset you can push to the Hugging Face Hub. That dataset is the artifact the clone and reinforce stages consume.

QuickstartCopy Anchor Link

uvx --from "so101-nexus[teleop]" so101-nexus teleop \
    --leader-port /dev/ttyACM0 \
    --seed 0

A Gradio UI opens in your browser. Most settings live in the UI; the CLI only takes connection-level options. For a repeat-use install of the teleop extra, see Installation.

Hardware and the serial portCopy Anchor Link

You need a physical SO-100 or SO-101 leader arm on USB and a host with access to its serial port. If the device is not writable, fix permissions and confirm the port:

sudo chmod 666 /dev/ttyACM0
lerobot-find-port

The Configure step reports leader-port reachability before you fill in the rest of the form, with a Recheck Port button, and blocks Initialize Session while the port is not ready. A connect-time failure keeps the session alive on the Initialize step with recovery guidance, so you can fix the port in another terminal and click Retry Initialization.

Session flowCopy Anchor Link

1. Configure. Pick the environment, robot type, episode count, FPS, camera resolutions, action space, countdown, dataset fields, Max Steps, and Success Hold.
2. Initialize. The app connects to the leader arm and creates the dataset.
3. Record. A countdown plays, then the simulation mirrors the leader arm. The live wrist feed renders in the UI.
4. Review. After each episode, the UI shows a synchronized replay of the available wrist, overhead, and side views, with camera labels and a joint-state plot.
5. Approve or discard. Accept to write the episode into the dataset, or re-record.
6. Push. When all episodes are done, check Hub Repo ID and push to the Hugging Face Hub.

The Complete step shows the original local recording directory and an editable Hub Repo ID. You can change the upload destination to your-username/dataset-name, including after an upload failure. The local recording stays in its original directory.

If you forgot your username or used the wrong namespace, correct Hub Repo ID on the Complete step. Then click Push to Hub again. You do not need to record the episodes again.

CLI argumentsCopy Anchor Link

Cameras and recorded fieldsCopy Anchor Link

Wrist, overhead, and side cameras are captured on every step and selected for the dataset by default. Wrist and overhead observations are wired automatically, and their width and height sliders set the image resolutions. The side view uses the overhead resolution and the simulator's render camera. Its azimuth is randomized between 105 and 115 degrees, and its elevation between -27 and -23 degrees, on each seeded reset. A config factory can override these ranges through RenderConfig, including equal bounds for a fixed view. The default side distance adapts to the sampled angles and spawn region. See automatic side framing for distance overrides.

Under Advanced Settings, Dataset fields, a checkbox group selects which fields are persisted; deselecting an optional field drops it from the declared feature schema and from every recorded frame. observation.state, action, reward, success, done, and task stay forced on because LeRobot datasets require them for policy training and task indexing.

Recorded dataset formatCopy Anchor Link

Datasets use the LeRobot v3 SO follower schema. Always present:

• observation.state: follower readback, 6D
• action: commanded absolute joint positions, 6D
• reward: scalar transition reward from the same environment step as action
• success and done: episode success and termination flags
• task: language description, from LeRobot v3 metadata

Optional, on by default: observation.images.wrist, observation.images.overhead, observation.images.side, and observation.environment_state (privileged low-dim state: TCP and object pose, grasp, offsets).

Body joints are stored in LeRobot degree units and the gripper in RANGE_0_100 percent, matching sim_so_follower and SO100/SO101 MolmoAct2 normalization. observation.state is simulated follower readback, not the leader echo.

On finalize, before any Hub push, the app writes the environment id and customization overrides to meta/so101_nexus_env.json inside the dataset. That file is itself a valid config profile, so --env-config-profile meta/so101_nexus_env.json reproduces a recording's environment. It captures UI and CLI overrides only; a separate --env-config-profile used at record time is not merged in.

For pick-and-place v2, both the teleop recorder and RolloutRecorder also save meta/placement_contracts/episode_NNNNNN.json. Each file stores the episode's resolved evaluator contract. The recorder refuses to overwrite an existing contract. It does not add these diagnostics to policy features or relabel legacy episodes. See the placement contract for benchmark migration rules.

The recorder saves meta/reproducibility/episode_NNNNNN.json for each approved episode. The effective episode seed is the base seed plus the approved episode index. A discarded episode keeps its seed when you record it again. Each file also stores the task description and the initial simulator state. The state includes robot and object positions, velocities, controls, mocap poses, camera transforms, and sampled colors. These snapshot fields preserve native simulator units for replay: hinge angles use radians, translations use meters, and camera FOV uses degrees. User-facing joint actions still use degrees.

On first recording the app creates a simulator-only calibration file at $HF_LEROBOT_CALIBRATION/robots/sim_so_follower/teleop_sim.json, or under the LeRobot default calibration root when that variable is unset. No physical follower calibration is needed.

Environment customizationCopy Anchor Link

Teleop uses the same typed config objects as the regular Gymnasium API. The recorder starts from the selected environment's default config, applies UI, profile, and factory overrides, then wires the camera observations required for recording.

Advanced Settings exposes the common options. Turn on Apply Environment Customization to enable them; leave it off to use the default config unchanged.

Options that do not apply to the selected environment are ignored: pick-object settings apply to PickConfig, pick-and-place colors to PickAndPlaceConfig, stack-cube colors to StackCubeConfig. Field names and defaults are in Config reference.

Config profilesCopy Anchor Link

Use a profile for a repeatable recording setup. Profiles may be JSON or TOML and are merged in this order, with later sections overriding earlier ones: flat top-level override keys, common, the matching task section (pick, pick_and_place, or stack_cube), then envs.<env_id>.

[common]
ground_colors = ["gray", "white"]
robot_colors = ["yellow"]
spawn_min_radius = 0.15
spawn_max_radius = 0.25
reset_settle_frames = 5

[pick]
n_distractors = 1
objects = [
  { type = "cube", color = "green" },
  { type = "ycb", model_id = "009_gelatin_box" },
  { type = "ycb", model_id = "032_knife" },
]

[pick_and_place]
cube_colors = ["red", "green"]
target_colors = ["blue"]

[stack_cube]
cube_a_colors = ["red", "orange"]
cube_b_colors = ["blue", "green"]

[envs.MuJoCoPickLift-v1]
spawn_angle_half_range_deg = 60.0

uv run so101-nexus teleop \
    --leader-port /dev/ttyACM0 \
    --env-config-profile teleop-profile.toml

The profile object schema matches the classes in Scene Objects. String specs such as cube:blue and ycb:009_gelatin_box work for built-in object choices. Mesh objects use mapping syntax because file paths may contain separator characters:

[[pick.objects]]
type = "mesh"
name = "custom widget"
collision_mesh_path = "/path/to/collision.stl"
visual_mesh_path = "/path/to/visual.obj"
mass = 0.02
scale = 1.0

Config factoriesCopy Anchor Link

For advanced setups, pass a Python factory with --env-config-factory my_project.teleop_configs:build_config. It receives (env_id, base_config) and may return a config object:

from so101_nexus import CubeObject, PickConfig, YCBObject


def build_config(env_id, base_config):
    return PickConfig(
        objects=[CubeObject(color="green"), YCBObject("009_gelatin_box")],
        n_distractors=1,
    )

It may also return a dict of gym.make kwargs. If the dict includes "config", teleop still wires the recording cameras after the factory runs.

Custom environmentsCopy Anchor Link

Custom environments must be registered with Gymnasium before teleop builds the dropdown. Import the registration module and list the environment ID:

uv run so101-nexus teleop \
    --leader-port /dev/ttyACM0 \
    --env-module my_project.envs \
    --extra-env-id CustomPick-v1

A custom environment should accept SO-100/SO-101 six-joint actions in the standard joint order, expose current qpos through _get_current_qpos(), and return camera observations compatible with the recording cameras. Teleop captures all three views even when a dataset field is deselected. Custom environments must support render_mode="rgb_array" and return an RGB side view from render() at the overhead image resolution. A config without RenderConfig must set that resolution itself. If the unwrapped environment exposes task_description, teleop records it as the required LeRobot task field.

Published demonstration datasetsCopy Anchor Link

TroubleshootingCopy Anchor Link