Teleoperation
Drive a simulated SO-101 follower with a physical leader arm and record LeRobot v3 demonstration datasets.
SO101-Nexus ships a Gradio teleop recorder. A physical SO-100 or SO-101 leader arm drives a simulated follower, and every approved episode is written to a LeRobot v3 dataset you can push to the Hugging Face Hub. That dataset is the artifact the clone and reinforce stages consume.
Quickstart
uvx --from "so101-nexus[teleop]" so101-nexus teleop \
--leader-port /dev/ttyACM0 \
--seed 0A Gradio UI opens in your browser. Most settings live in the UI; the CLI only takes
connection-level options. For a repeat-use install of the teleop extra, see
Installation.
Policy correction collection
Run a trained policy in the same viewer and take control when it needs correction:
uv run --extra teleop so101-nexus teleop \
--leader-port /dev/ttyACM0 \
--policy-path outputs/train/checkpoints/last/pretrained_model \
--policy-fps 30 \
--policy-device cpuLeRobot checkpoints must include their preprocessor and postprocessor files.
The adapter loads the saved normalization statistics and resets cached actions on each handoff.
Set camera resolutions to match the checkpoint, and use its trained joint order and units.
Set the UI FPS to the checkpoint's action rate. Use --policy-fps to declare that rate
when it is absent from the checkpoint; initialization rejects mismatched rates.
Policy collection configures physics to advance exactly 1 / FPS simulated seconds per
executed action. Physics substeps divide that interval exactly and remain no larger
than the configured physics timestep. Wall-clock inference latency can slow collection,
but it does not change simulated action spacing or dataset timestamps.
For chunked models, each select_action call consumes one action at that rate.
The model and its processor pipelines retain responsibility for chunk replay and history.
The default policy output is an absolute six-joint target.
For a checkpoint trained to predict deltas from current readback, add
--policy-action-space joint_pos_delta.
This flag describes model output; the UI action-space setting describes saved dataset actions.
MolmoAct2 SO100/101
The original allenai/MolmoAct2-SO100_101 package loads directly:
uv run --extra teleop --extra molmoact2 so101-nexus teleop \
--policy-path allenai/MolmoAct2-SO100_101 \
--policy-device cudaSee Installation for model dependencies.
The loader selects continuous inference, the so100_so101_molmoact2 normalization tag,
the checkpoint's gripper normalization, and LeRobot's documented legacy joint-frame
conversion for this package. Shoulder lift reverses direction about 90 degrees;
elbow flex shifts by 90 degrees. Recorded states and actions stay in follower coordinates.
The published rollout uses 30 Hz;
keep the UI FPS at 30. Overhead and side RGB views are supplied by default, matching
the published example's top and side views. The native image processor handles resizing.
For another original MolmoAct2 checkpoint, provide --policy-norm-tag and --policy-fps.
Its normalization metadata must describe SO101 joint order and absolute joint pose control.
Loading support does not establish task success or cross-calibration compatibility.
The default conversion follows LeRobot's SO100/101 migration recipe; a custom calibration
can require different signs or offsets. A custom factory can configure LeRobot's
joint_signs and joint_offsets and load its matching processors. Preserve the
training frame when fine-tuning.
See LeRobot backward compatibility.
DM0.5 SO101 Pick Cube
Load the published SO101 checkpoint directly through LeRobot:
uv run --extra teleop --extra dm05 so101-nexus teleop \
--policy-path Dexmal/DM05-SO101-Pick-Cube \
--policy-device cuda \
--policy-fps 30Set the UI FPS to the training dataset's action rate, matching --policy-fps.
The Hub ID selects DM0.5 automatically. Hugging Face downloads and caches the base model,
LoRA weights, tokenizer, and normalization statistics on first use. No separate inference
server or OpenDM installation is required. See
Installation for dependencies and pip setup.
The native adapter preserves the release's SO101 prompt and head/wrist camera order.
Overhead RGB supplies the head view, and wrist RGB supplies the wrist view.
LeRobot normalizes observations, denormalizes output with the checkpoint's norm_stats.json,
and converts relative joint predictions to absolute targets. Each 50-action chunk keeps
its state reference from the observation that generated it. The five arm channels use
degrees, and the gripper uses percent. Keep --policy-action-space joint_pos.
Takeover and episode resets discard queued actions and the previous state reference.
For a local copy of this raw release, use --policy-type dm05 --policy-path /path/to/checkpoint.
Converted six-joint LeRobot DM0.5 checkpoints use the standard saved-processor loading path.
The generic lerobot/dm05_base release has a 14-dimensional action contract and cannot
drive the six-joint follower without SO101 fine-tuning.
To change viewpoints, map exactly two collector cameras to images_1 (head) and
images_2 (wrist), for example --policy-camera side=images_1 --policy-camera wrist=images_2.
Loading support does not establish task success or calibration compatibility.
Use a custom adapter for checkpoints with different units, joint frames, or action semantics.
If you already run OpenDM's SO101 server,
connect with --policy-type dm05 --policy-path http://127.0.0.1:7891/v1/infer --policy-fps 30.
The server selects and loads its own weights; --policy-device controls native loading only.
Other SO101 policy checkpoints
The loader uses LeRobot's registered policy classes rather than a model allowlist. LeRobot 0.7 checkpoints for ACT, Diffusion, SmolVLA, and other registered families can use the same path when they have six SO101 joint channels, matching units and action semantics, saved processor pipelines, and the appropriate optional model dependencies. A policy family name alone does not establish checkpoint compatibility. See Installation for dependencies.
Use repeatable camera mappings when checkpoint names differ:
uv run --extra teleop so101-nexus teleop \
--policy-path /path/to/so101/pretrained_model \
--policy-fps 30 \
--policy-camera overhead=front \
--policy-camera wrist=wristMappings rename collector inputs before preprocessing. Unmapped cameras remain available. For original MolmoAct2 packages without camera-name metadata, supplied mappings select the views passed to the model. Check the checkpoint's expected viewpoints as well as names. Models with end-effector actions, padded outputs, or another robot's joint order need a custom adapter rather than being accepted based on their action dimension alone.
Intervention controls
- Click Start policy episode. The policy controls the follower after the countdown.
- Click Pause and take control (T) when correction is needed. Policy stepping stops immediately. The leader moves gradually to the follower's current pose, including the inverse wrist roll offset. Keep the leader clear while Paused: aligning leader is shown.
- When alignment finishes or pauses, hold the physical arm and gripper. Click Start correction (C) to release the motors and begin correction. The reminder below the button replaces a separate confirmation checkbox. If alignment needs attention, the button reads Take control from this pose (C). The first correction action preserves the current simulated pose. Your movements from this point drive the follower, with pose differences compensated for every joint and the gripper. The simulation stays paused until you start correction.
- Click Resume policy (P) to continue from the corrected state with a fresh prediction.
- Repeat as needed, then review and approve the entire episode.
Pause (H) freezes simulation stepping and cancels an active leader synchronization. Press each shortcut once; you do not need to hold it. After pausing, use T to align the leader again before starting a correction, or P to resume the policy. The controller panel shows remaining simulation time and steps. Paused time does not consume this budget. Episodes end at Max Steps or after task success; Stop and review episode ends recording and opens review. The keyboard shortcuts work while the Record controls are visible and you are not typing in a field. Slow model inference does not block a takeover request. A request takes effect between simulator steps. Late predictions and cached chunks from before a handoff cannot control the follower afterward. Simulation time freezes while paused or while inference or synchronization is pending; these waits add no dataset frames. This collector currently supports the simulated follower, not real-robot deployment.
Each Start policy episode returns the leader to the configured resting tool
pose before the countdown. LeRobot's RobotKinematics solves the five arm joint
targets from the measured leader pose using RobotConfig.rest_qpos_deg as the
resting tool reference in the leader's joint frame. A zero-degree wrist rest
target keeps the physical leader wrist at zero; the teleoperation wrist offset
is applied when aligning with or resetting the follower, not when selecting
the leader's resting pose. The gripper uses its configured resting angle, clamped
to the simulator's joint limits. You do not need to position the leader before
initializing. Motors ramp to the target over four seconds with a gentle start
and stop. Keep the arm clear during the return.
At episode completion, including Stop and review episode, the leader
automatically returns to rest before Review becomes ready. This return preserves
the recorded frames and simulator state. Starting another episode checks the
resting pose again before its countdown. Pressing Stop while the arm is already
returning cancels that return and holds the measured pose. Recording errors and
process shutdown hold the leader without starting an automatic return.
A failed IK solve, return timeout, or motor error stops preparation before the
countdown or policy inference. Policy correction shows the failure and joint
readings. A return timeout means a measured joint did not reach its target within
the four-second ramp plus three seconds for settling. Check the listed joint
readings before retrying. Commands outside the leader's calibrated motor ranges
are rejected before movement.
The simulator resets from the measured leader pose after the return, so manual
repositioning between episodes is unnecessary. Ordinary teleoperation keeps its
current-pose reset behavior.
Synchronization uses the calibrated leader motor bus, with body joints in degrees and the gripper in percent.
Takeover alignment ramps at up to 20 degrees per second for body joints and 40
percentage points per second for the gripper. The episode return uses the timed
ramp described above. The reset solver uses the bundled SO-101 URDF and matching
TCP frame; custom environments must expose a robot configuration and joint bounds.
Before holding, each measured joint target must be acknowledged by that motor before
its torque is enabled. If a motor fails, holding is still attempted for the other joints;
the overall hold remains unconfirmed.
Default alignment tolerances are 10 degrees for body joints and 5 percentage points
for the gripper. Set --leader-alignment-tolerance-deg and
--leader-gripper-tolerance-percent to override them. Both must be positive and finite.
These tolerances guide alignment readiness; they do not block supported takeover
from a paused alignment pose.
Wrist roll uses circular angular distance, and its inverse offset wraps into the physical
single-turn range. During correction, wrist targets use the equivalent angle nearest
follower readback so crossing the angle boundary does not flip the simulated target.
An alignment timeout holds the measured physical pose and keeps the episode paused.
The controller panel lists the joints that did not reach their targets, including measured
and target positions. Hold the arm and gripper, press C, and take control from
this pose. You can also press T to retry alignment or P to resume the policy.
Torque changes and synchronization reads and transmissions retry communication failures
up to three times through LeRobot.
A persistent motor communication error that prevents holding still ends recording.
Pausing, finishing an episode, and process cleanup also preserve motor torque.
Recording errors stay visible in Review, along with any failure to hold the leader.
If holding is unconfirmed, support the arm and gripper immediately.
These are target-command bounds, not guarantees of physical motor speed or successful holding
when motor communication or power fails.
To release the leader after recording stops, expand Leader motors inside Policy correction. Support the arm and gripper, check the confirmation, and click Release motors. Policy correction stays available in Review and Complete. Finalization and reinitialization require this explicit release. The confirmation resets after each handoff or release. Closing the process without releasing the leader preserves motor torque; support and park the arm before disconnecting power.
Custom model adapters
Use --policy-factory my_project.policy:build_policy for a custom model or remote inference client.
The no-argument factory returns an object with select_action(batch) and reset() methods.
This runnable scaffold holds the current pose; replace select_action with your model's prediction:
class MyPolicy:
def select_action(self, batch):
return batch["observation.state"].copy()
def reset(self):
pass
def build_policy():
return MyPolicy()The batch contains an unbatched float32 observation.state, task text, and available
observation.images.wrist, observation.images.overhead, and observation.images.side arrays.
When available, observation.environment_state contains the privileged component vector.
Images use HWC uint8 RGB format. The state follows SO101_JOINT_NAMES order.
Body joints use degrees and the gripper uses 0 to 100 percent, matching follower readback.
Return a finite NumPy array or torch tensor of shape (6,) in the same order and units.
For delta output, use the policy action-space flag described above.
Map camera names, normalization, and any other model-specific inputs inside the adapter.
reset() must clear action chunks and recurrent history. All model calls run serially on an inference thread.
Custom factories own device selection; --policy-device applies to the built-in LeRobot loader.
A factory adapter may expose an integer fps; otherwise declare its rate with --policy-fps.
For another input device, InterventionSession.request("human"), request("hold"),
and request("policy") expose the same handoffs independently of Gradio.
The session accepts a synchronizer with start(target, duration_s=None), tick(), cancel(), and release()
methods and a holding property. cancel() stops alignment and holds the measured pose;
tick() keeps torque enabled after convergence. Only release() disables torque.
For a staged takeover, call request("human", pause_after_sync=True), then
start_correction() after the operator supports the leader. It captures an anchor
before releasing torque, so correction starts from the current follower pose even
when alignment timed out. The anchor resets at each handoff. A direct request("human")
aligns and releases the leader, so its caller must obtain support confirmation first.
end_episode() holds the leader; release_leader() explicitly releases an idle leader.
start_parking(target) begins a four-second return using follower-coordinate targets.
Advance it with synchronizer.tick(), then call end_episode() to finish or cancel.
SOLeaderSynchronizer provides the SO leader implementation.
These public objects are available from so101_nexus.
Training on corrections
Collection records autonomous context and human corrections in one continuous episode. Dataset fields identify which actions are expert supervision; the schema is described below. Do not train behavior cloning on autonomous actions as if they were human demonstrations. Aggregate the original demonstrations with human-controlled targets from the new episodes.
For chunked policies, sample human-controlled starting frames and mask targets after the human segment ends. The public helper works with NumPy arrays and torch tensors:
import torch
from so101_nexus import intervention_chunk_mask
intervention_ids = torch.tensor([0, 1, 1, 0, 2])
mask = intervention_chunk_mask(intervention_ids, horizon=3)Pass one episode at a time. Each mask row describes valid human targets for a chunk starting at that frame. Zero marks autonomous frames; positive IDs distinguish correction segments. Use the mask in your trainer's loss, and exclude autonomous starting frames from correction sampling. The standard LeRobot training CLI does not automatically apply this helper or filter these fields. Recovery and correction share the same human label in this initial collector.
Hardware and the serial port
You need a physical SO-100 or SO-101 leader arm on USB and a host with access to its serial port. If the device is not writable, fix permissions and confirm the port:
sudo chmod 666 /dev/ttyACM0
lerobot-find-portThe Configure step reports leader-port reachability before you fill in the rest of the form,
with a Recheck Port button, and blocks Initialize Session while the port is not ready. A
connect-time failure keeps the session alive on the Initialize step with recovery guidance, so
you can fix the port in another terminal and click Retry Initialization.
Session flow
- Configure. Pick the environment, robot type, and episode count. Leave the dataset destination blank for local recording. Open Recording settings to change FPS, camera resolutions, action space, countdown, dataset fields, Max Steps, or Success Hold.
- Initialize. The app connects to the leader arm and creates the dataset.
- Record. A countdown plays, then the simulation mirrors the leader arm. The live wrist, overhead, and side feeds render together in the UI.
- Review. After each episode, the UI shows a synchronized replay of the available wrist, overhead, and side views. Expand Joint states and rewards for diagnostics.
- Save or retry. Click Save episode to write it to the local dataset, or Discard and retry to record another attempt. Start the next episode when ready.
- Finish. Click Finish session to finalize the local dataset and disconnect the leader. For policy collection, first support the arm and release its motors under Leader motors. To upload, open Upload to Hugging Face (optional), check Hub Repo ID, and click Push to Hub.
The simulator stays loaded between episodes and resets with the current leader pose and episode seed. Sessions with a config profile or factory rebuild the simulator each episode so configuration changes still take effect.
The Complete step shows the original local recording directory. The upload section contains
an editable Hub Repo ID.
You can change the upload destination to your-username/dataset-name, including after an upload failure.
The local recording stays in its original directory.
If you forgot your username or used the wrong namespace, correct Hub Repo ID on the Complete step. Then click Push to Hub again. You do not need to record the episodes again.
CLI arguments
| Argument | Type | Default | Description |
|---|---|---|---|
--leader-port | str | /dev/ttyACM0 | Serial port of the leader arm |
--leader-id | str | so101_leader | Device identifier for the leader arm |
--wrist-roll-offset-deg | float | -90.0 | Wrist roll offset in degrees |
--policy-path | str | None | None | LeRobot checkpoint directory or Hub model ID |
--policy-factory | str | None | None | No-argument Python module:function returning a model adapter |
--policy-type | str | None | None | Optional checkpoint type check or original MolmoAct2 loader selection |
--policy-norm-tag | str | None | None | Original MolmoAct2 normalization tag; inferred for the published SO100/101 package |
--policy-fps | int | None | None | Required action rate when unavailable from model metadata; 30 for the published MolmoAct2 SO100/101 rollout |
--policy-camera | repeatable str | [] | Camera mapping such as overhead=front or wrist=wrist |
--policy-device | str | cpu | Device for the built-in LeRobot loader |
--policy-action-space | str | joint_pos | Model output: absolute joint_pos or readback-relative joint_pos_delta |
--seed | int | 0 | Base seed for simulation episodes |
--env-config-profile | str | None | None | JSON or TOML profile with environment config overrides |
--env-config-factory | str | None | None | Python module:function that returns a config or gym.make kwargs |
--env-module | repeatable str | [] | Import a module that registers custom Gymnasium environments |
--extra-env-id | repeatable str | [] | Add a custom environment ID to the UI dropdown |
Cameras and recorded fields
Wrist, overhead, and side cameras are captured on every step and selected for the dataset by
default. Live previews update at up to 10 Hz; recorded frames use the configured FPS.
Wrist and overhead observations are wired automatically, and their width and height
sliders set the image resolutions. The side view uses the overhead resolution and the
simulator's render camera. Its azimuth is randomized between 105 and 115 degrees, and its
elevation between -27 and -23 degrees, on each seeded reset. A config factory can override
these ranges through RenderConfig, including equal bounds for a fixed view.
The default side distance adapts to the sampled angles and spawn region.
See automatic side framing for distance overrides.
Under Recording settings, the Dataset fields checkbox group selects which fields are
persisted; deselecting an optional field drops it from the declared feature schema and from
every recorded frame. observation.state, action, reward, success, done, and task
stay forced on because LeRobot datasets require them for policy training and task indexing.
Recorded dataset format
Datasets use the LeRobot v3 SO follower schema. Always present:
observation.state: follower readback, 6Daction: executed joint targets, 6D (absolute by default)reward: scalar transition reward from the same environment step asactionsuccessanddone: episode success and termination flagstask: language description, from LeRobot v3 metadata
Policy correction datasets additionally require is_intervention (float32, 0 or 1) and
intervention_id (int64, 0 for autonomy, positive per human segment).
The recorder preserves the executed follower command after limiting, paired with pre-action readback.
When the UI selects joint_pos_delta, correction datasets store target minus pre-action readback.
Ordinary teleop delta datasets retain their frame-to-frame target differences.
Episode reproducibility metadata includes the policy path or factory, device, camera mappings,
normalization options, effective FPS, and policy action convention.
A local checkpoint path identifies its source; it does not capture a hash of mutable weights.
Optional, on by default: observation.images.wrist, observation.images.overhead,
observation.images.side, and
observation.environment_state (privileged low-dim state: TCP and object pose, grasp,
offsets).
Body joints are stored in LeRobot degree units and the gripper in RANGE_0_100 percent,
matching sim_so_follower and SO100/SO101 MolmoAct2 normalization. observation.state is
simulated follower readback, not the leader echo.
On finalize, before any Hub push, the app writes the environment id and customization
overrides to meta/so101_nexus_env.json inside the dataset. That file is itself a valid
config profile, so --env-config-profile meta/so101_nexus_env.json
reproduces a recording's environment. It captures UI and CLI overrides only; a separate
--env-config-profile used at record time is not merged in.
For pick-and-place v2, both the teleop recorder and RolloutRecorder also save
meta/placement_contracts/episode_NNNNNN.json.
Each file stores the episode's resolved evaluator contract.
The recorder refuses to overwrite an existing contract.
It does not add these diagnostics to policy features or relabel legacy episodes.
See the placement contract for benchmark migration rules.
The recorder saves meta/reproducibility/episode_NNNNNN.json for each approved episode.
The effective episode seed is the base seed plus the approved episode index.
A discarded episode keeps its seed when you record it again.
Each file also stores the task description and the initial simulator state.
The state includes robot and object positions, velocities, controls, mocap poses, camera transforms, and sampled colors.
These snapshot fields preserve native simulator units for replay: hinge angles use radians,
translations use meters, and camera FOV uses degrees. User-facing joint actions still use degrees.
For environments that expose rendering metadata, the top-level rendering field records
camera quality and allocated renderer details after the recording finishes.
RolloutRecorder saves the same sidecar path with seed and rendering fields when it writes a dataset.
It also returns the metadata in EpisodeResult.info["rendering"] without a dataset.
Lazy visualization cameras appear after their first render.
See RenderingQuality for profile selection and hardware limits.
On first recording the app creates a simulator-only calibration file at
$HF_LEROBOT_CALIBRATION/robots/sim_so_follower/teleop_sim.json, or under the LeRobot default
calibration root when that variable is unset. No physical follower calibration is needed.
Environment customization
Teleop uses the same typed config objects as the regular Gymnasium API. The recorder starts from the selected environment's default config, applies UI, profile, and factory overrides, then wires the camera observations required for recording.
Open Environment customization and turn on Apply Environment Customization to reveal the supported options. Leave it off to use the default config unchanged. Max Steps and Success Hold (s) are under Recording settings.
| Option | Effect |
|---|---|
Pick Object Pool | Built-in cube and YCB objects for pick tasks |
Minimum Distractors, Maximum Distractors | Inclusive count bounds for tasks that support distractors. Equal values give a fixed count; see sampling and pool limits |
Ground Colors, Robot Colors | Sampled per reset when multiple colors are selected |
Spawn Min Radius, Spawn Max Radius, Spawn Angle Half Range (deg) | Spawn-region controls shared by supported tasks |
Reset Settle Frames | No-op frames after reset before the first recorded frame (default 5), applied before cameras are wired |
Pick-and-Place Cube Colors, Pick-and-Place Target Colors | Sampled cube and target colors |
Stack Cube A Colors, Stack Cube B Colors | Sampled colors for the two cubes |
Max Steps | Overrides the episode's step cap |
Success Hold (s) | Seconds to keep recording after success before auto-stopping (default 0.5, 0 stops immediately) |
Options that do not apply to the selected environment are ignored: pick-object settings apply
to PickConfig, pick-and-place colors to PickAndPlaceConfig, stack-cube colors to
StackCubeConfig. Field names and defaults are in Config reference.
Config profiles
Use a profile for a repeatable recording setup. Profiles may be JSON or TOML and are merged in
this order, with later sections overriding earlier ones: flat top-level override keys,
common, the matching task section (pick, pick_and_place, or stack_cube), then
envs.<env_id>.
[common]
ground_colors = ["gray", "white"]
robot_colors = ["yellow"]
spawn_min_radius = 0.15
spawn_max_radius = 0.25
reset_settle_frames = 5
[pick]
min_distractors = 0
max_distractors = 2
objects = [
{ type = "cube", color = "green" },
{ type = "ycb", model_id = "009_gelatin_box" },
{ type = "ycb", model_id = "032_knife" },
]
[pick_and_place]
cube_colors = ["red", "green"]
target_colors = ["blue"]
[stack_cube]
cube_a_colors = ["red", "orange"]
cube_b_colors = ["blue", "green"]
[envs.MuJoCoPickLift-v1]
spawn_angle_half_range_deg = 60.0uv run so101-nexus teleop \
--leader-port /dev/ttyACM0 \
--env-config-profile teleop-profile.tomlThe profile object schema matches the classes in Scene Objects. String
specs such as cube:blue and ycb:009_gelatin_box work for built-in object choices. Mesh objects
use mapping syntax because file paths may contain separator characters:
[[pick.objects]]
type = "mesh"
name = "custom widget"
collision_mesh_path = "/path/to/collision.stl"
visual_mesh_path = "/path/to/visual.obj"
mass = 0.02
scale = 1.0Config factories
For advanced setups, pass a Python factory with
--env-config-factory my_project.teleop_configs:build_config. It receives
(env_id, base_config) and may return a config object:
from so101_nexus import CubeObject, PickConfig, YCBObject
def build_config(env_id, base_config):
return PickConfig(
objects=[CubeObject(color="green"), YCBObject("009_gelatin_box")],
n_distractors=1,
)It may also return a dict of gym.make kwargs. If the dict includes "config", teleop still
wires the recording cameras after the factory runs.
Custom environments
Custom environments must be registered with Gymnasium before teleop builds the dropdown. Import the registration module and list the environment ID:
uv run so101-nexus teleop \
--leader-port /dev/ttyACM0 \
--env-module my_project.envs \
--extra-env-id CustomPick-v1A custom environment should accept SO-100/SO-101 six-joint actions in the standard joint order,
expose current qpos through _get_current_qpos(), and return camera observations compatible
with the recording cameras. Teleop captures all three views even when a dataset field is
deselected. Custom environments must support render_mode="rgb_array" and return an RGB
side view from render() at the overhead image resolution. A config without RenderConfig
must set that resolution itself. If the unwrapped environment exposes task_description,
teleop records it as the required LeRobot task field.
Published demonstration datasets
| Task | Dataset | Episode viewer |
|---|---|---|
| PickLift | johnsutor/MuJoCoPickLift-v1 | episode 0 |
| PickAndPlace | johnsutor/MuJoCoPickAndPlace-v1 | episode 0 |
Robot identification
Launch the local Gradio app from the repository root:
uv run --extra teleop --extra sysid python -m so101_nexus.calibrationThe installation guide describes the required extras. The first milestone is a saved, physically aligned profile. Motion collection and dynamics fitting are optional afterward. One shared session owns the hardware. Browser reconnects show that same session. Port ownership prevents overlapping workers in this app. Close other applications that use the follower port.
Connect and align the reference
- Select Connect robot. The port defaults to
/dev/ttyACM0. - Support the arm with motor torque off.
- Match the highlighted physical joint to the cyan reference.
- Select Matches, capture and continue for each of the five body joints.
- Compare the closed and open physical jaw endpoints with their reference views.
- Confirm the jaw endpoints, then select Save profile.
- Download the profile directly from Your profile.
Saving an edited profile creates a new filename and preserves the original. The saved-profile selector lists available profiles, so typing a path is unnecessary for the usual setup.
The robot view stays upright and fills most of the screen. Front, Side, Top, and Fit view change the camera only. The guide opens a wrist close-up for the bend and twist steps. Visible quarter-turn buttons preview a correction to the measured wrist mapping without moving hardware. A preview clears that joint's confirmation. Physical reference capture is still required.
Connection and saved profile contains serial discovery, the calibration directory, and trusted profile loading. Known reference angles, jaw geometry, save paths, and timing remain in contextual advanced sections.
Discovery and live measurements do not enable torque, write calibration, or send movement commands. The viewport shows a translucent reference robot and a distinct measured robot in one coordinate frame. The persistent legend identifies both robots. View controls change the camera only. Reference controls change the simulated reference only.
Encoder calibration does not establish the physical model zero. Before reference capture, the measured overlay uses provisional offsets and cannot prove alignment. A capture associates the live encoder reading with a physical joint position that the operator understands. An arbitrary pose cannot establish that reference. The app captures encoder readings directly, so angle transcription is unnecessary.
Body-joint alignment and gripper endpoint mapping are separate steps. The profile reference defines the units, endpoint mapping, and saved fields.
The app opens the profile pane when it finds a saved Nexus profile in persistent application storage. Saved profiles are available regardless of the directory used to launch the app. Select Connect saved robot to verify the profile against the follower's motor calibration. The app finds a profile with matching motor calibration and restores its completed alignment without new reference captures. Recalibrate robot starts again at the first joint and preserves the saved file. Connect the robot before starting recalibration. Use Import profile in Connection and saved profile to select a profile from another machine. Download the aligned or tuned profile to export it. Older calibration files are copied automatically when found; their originals are preserved. Advanced storage and migration can import an older calibration folder from another location. Storage defaults and environment overrides are listed in Calibration storage. LeRobot encoder calibration alone does not establish physical model alignment.
Prepare dynamics calibration
After saving the profile, select Dynamics tuning. The main flow is Prepare robot, Record calibration, then Tune simulation. You can stop after saving if you only need aligned joint coordinates.
- Support the arm at a raised pose with room for motion.
- Select Arm supported, prepare robot.
- Keep supporting the arm until it holds steadily.
Preparation reads the live pose, checks all three motions in simulation, then enables holding torque. The robot does not start the recording until the separate hands-clear action. A visible activity indicator covers motion checks and other slow actions. An indeterminate indicator remains visible while hardware prepares to hold. Stop commands stays beside the current action. During the motion check, it cancels preparation without enabling hold. The run plan shows all three motions and their total duration before preparation. Two recordings tune the simulator. The third is an independent check, recorded in the same sequence. No separate held-out collection is needed.
Motion settings and workspace contains manual centers, motion settings, and a preview without holding torque.
The preview and collector use the same trajectory configuration. A change to the alignment or motion configuration invalidates the previous preflight. Editing motion settings during holding or collection stops further commands. Repeat preflight before restarting. The app generates a multisine, sequential joint chirps, and an independent held-out trajectory. Smooth envelopes start and end each episode at the center pose. Velocity and acceleration limits bound the commands.
The preflight evaluates kinematic and dynamic MuJoCo rollouts before hardware collection.
It examines physical calibration ranges, model joint limits, collisions, and tool clearance.
The default scene contains the robot and a horizontal table.
Simulation cannot represent unmodeled obstacles or establish hardware safety.
For a custom workspace model, use collect_excitation(..., model=workspace_model) through the library API.
Record dynamics calibration
- Wait until the supported arm holds steadily.
- Remove your hands from the arm and its workspace.
- Select Hands clear, record calibration.
One start records all three motions automatically, including the independent check. The run plan shows the current motion and how many recordings are saved. After all three recordings finish, the app opens Tune simulation automatically. A stopped or incomplete sequence shows recovery instructions. Reconnect the same profile to keep a completed dataset available without recording again. A changed joint mapping or command period requires matching recordings before tuning; previous files remain available.
The driver briefly disables torque during connection. An unsupported arm can fall. The app transfers serial ownership from the monitor to one collection worker. The collector requires matching calibration, hardware PID coefficients, and a follower without cameras. An initial pose mismatch, excessive tracking error, slow reads, or excessive command lateness stops collection. The collector tolerates brief scheduler pauses before reading a fresh pose and resuming. Motor reads and writes retain stricter timing limits. The GUI runs collection in a separate process, so UI redraws and garbage collection do not pause the motor loop. An independent collector watchdog stops new commands if the UI stops responding. It never sends a burst of commands to catch up. Actual timestamps remain part of each recording. A timing error reports the measured delay, allowed delay, run, and sample. Collection failures appear as a top-right warning and a persistent error beside the recovery action. The collector does not command a return motion after an error.
Stop commands ends further commands and preserves torque at the last commanded target. Disconnect closes the serial connection and preserves the current torque state. Neither action releases the motors. If browser heartbeats stop for five seconds, the worker cancels collection and preserves torque. A stopped or disconnected arm can still hold its previous target.
To release the motors, support the arm first. Then select Release motors in the app. This action disables torque, so an unsupported arm can fall.
After a stopped attempt, select Reconnect and prepare another attempt. If torque remains enabled, support the arm and use the explicit release control before repeating the motion check. A missing orange overlay means that live measurements are unavailable. The app explains how to reconnect.
Each completed episode saves immediately to a new output directory.
The app chooses a numbered directory for each retry and preserves previous recordings.
An unsuccessful attempt removes its new output directory only if that directory remains empty.
Each directory with completed recordings includes the exact collection profile in source_profile.json.
The fitter uses this snapshot, even after later profile edits.
Recordings contain sent commands, read intervals, host write times, and a digest of the profile coordinate contract.
Host write times do not measure motor receipt.
RecordedTrajectory.load(path) reads these files without pickle.
Fit and evaluate dynamics
Select Tune simulation after collection.
The app reports whether the independent check improved and provides the candidate profile for download.
If the check does not improve consistently, the app recommends the original profile.
Detailed results shows prediction errors and parameter diagnostics.
The fitter uses training episodes for optimization and separate held-out episodes for evaluation.
It writes candidate_profile.json and metrics.json.
The candidate contains effective simulation dynamics, not hardware register gains.
The app provides the resulting files for download and shows their local locations.
It never selects candidate dynamics automatically.
Review the held-out errors, parameter bounds, confidence intervals, and correlations before selecting a profile. Weak excitation can leave parameters poorly identified, even with low training error. Confidence intervals assume independent residuals and can underestimate uncertainty for correlated trajectories. A lower held-out RMSE does not establish successful policy transfer.
For explicit parameter selection and freezing, use the library API:
from so101_nexus import RecordedTrajectory, RobotProfile, fit_robot_profile
profile = RobotProfile.load("my_robot_profile.json")
recordings = [
RecordedTrajectory.load(f"sysid_recordings/{index:02d}_{name}.npz")
for index, name in enumerate(("multisine", "joint_chirps", "heldout"))
]
result = fit_robot_profile(
profile,
recordings,
parameters=("shoulder_pan.kp", "shoulder_pan.kd"),
freeze=("shoulder_pan.kd",),
output_dir="sysid_fit_selected",
)The default fit selects proportional and derivative gains for all six joints.
Other supported parameters are armature and frictionloss.
Passive damping and unselected parameters retain their baseline values.
Reuse the selected profile
Load the reviewed candidate through RobotConfig for training on either backend. Use the same profile with the simulator follower:
from pathlib import Path
from so101_nexus.lerobot_adapter import SimSOFollower, SimSOFollowerConfig
robot = SimSOFollower(
SimSOFollowerConfig(
id="my_robot",
env_id="MuJoCoMove-v1",
profile_path=Path("sysid_fit/candidate_profile.json"),
)
)Keep the hardware PID coefficients, payload, and command frequency consistent with collection. Joint-trajectory identification does not calibrate camera geometry or establish grasp-contact fidelity.
Comparison with Playground
Nexus preserves LeRobot calibration files and adds explicit model alignment in a separate profile. Playground stores an explicit reference zero and uses modular encoder differences. Its encoder conversion uses 4096 counts per turn. LeRobot degree normalization uses 4095. Playground calibration files require an explicit conversion before reuse in Nexus. They cannot replace LeRobot calibration files directly. See the Playground calibration implementation.