Observations

Every SO101-Nexus environment builds its observation from a list of observation components. Components are lightweight descriptor classes that tell the environment which data to include. State components contribute slices of a flat vector; camera components add image tensors to a dictionary observation.

Observation ComponentsCopy Anchor Link

State ComponentsCopy Anchor Link

State components produce fixed-size slices of the observation vector.

GazeDirection and GazeState are anchored at the wrist camera, not at the gripper tip. Tasks without a target object (Move) do not support either component.

Rest target observationsCopy Anchor Link

PickReturn includes RestingJointPositions, TargetPosition, and TargetOffset by default. The first contains the five configured arm angles in degrees, following RobotConfig.rest_qpos_deg. The existing JointPositions component reports current angles in radians; convert units before comparing them. TargetPosition gives the world-frame TCP position at the rest posture in meters. TargetOffset gives that position minus the current TCP position. The rest target stays fixed when reset randomizes the starting posture.

This makes both the joint target and the relative end-effector target observable, similar to Habitat's relative resting-position sensor. In obs_mode="visual", the existing actor state remains six joint positions; the configured rest components enter info["privileged_state"] with other state components. To expose rest targets directly to a camera-based actor, use obs_mode="state" and include the desired rest and camera components in observations.

Camera ComponentsCopy Anchor Link

Camera components add image tensors to a dict-style observation space. They have size = 0 in the state vector.

Camera components accept width and height parameters (default 640x480). WristCamera also supports domain randomization parameters for FOV and pitch.

from so101_nexus import WristCamera, OverheadCamera

# Default resolution
wrist = WristCamera()

# Custom resolution with FOV randomization
wrist = WristCamera(width=224, height=224, fov_deg_range=(60.0, 90.0))

# Overhead camera
overhead = OverheadCamera(width=320, height=240, fov_deg=45.0)

Both backends render camera observations with the same keys, shapes, and dtypes, except for Warp's leading batch dimension. MuJoCo uses a separate OpenGL renderer for each environment. Warp traces rays for all worlds in one batch on the simulation device. The renderers produce different pixel values at the same simulator state. See Backends for execution and render modes.

Depth observationsCopy Anchor Link

Select depth on either camera component:

from so101_nexus import JointPositions, OverheadCamera, TouchConfig, WristCamera

config = TouchConfig(observations=[
    JointPositions(),
    WristCamera(width=224, height=224, modalities=("rgb", "depth")),
    OverheadCamera(width=320, height=240, modalities=("depth",)),
])

Use this config with either backend's environment creation loop. The observation contains state, wrist_camera, wrist_camera_depth, and overhead_camera_depth. The camera API defines the accepted modalities and defaults.

Depth measures distance along the camera's forward axis, rather than Euclidean distance along each pixel's ray. A flat surface parallel to the image plane has constant depth. RGB and depth share the camera pose, resolution, and field of view, including wrist randomization at reset. Warp depth stays on the simulation device. Returned images retain their values across later steps and resets.

Missing surfaces and surfaces beyond the far clipping plane return the far distance, model.vis.map.zfar * model.stat.extent. For Warp, the host model is envs.unwrapped.mjm. Depth values are not normalized, so distances greater than one meter remain greater than one. Rasterization and ray tracing can disagree at silhouettes and near clipping boundaries. MuJoCo clips surfaces at its near plane. Warp can report hits closer than that plane.

Depth arrays are numeric observations. The LeRobot camera adapter, RGB image processors, EnvHub wrapper, and video recorders retain their RGB interfaces. These interfaces do not record or prepare depth for policies automatically.

Composing ObservationsCopy Anchor Link

Pass a list of components via the observations parameter on any config. Each task provides sensible defaults when observations is not specified.

from so101_nexus import (
    PickConfig, JointPositions, JointVelocities, EndEffectorPose, GraspState,
    ObjectPose, ObjectOffset, WristCamera,
)

# State-only observations
config = PickConfig(observations=[
    JointPositions(),
    JointVelocities(),
    EndEffectorPose(),
    GraspState(),
    ObjectPose(),
    ObjectOffset(),
])

# Add a wrist camera to the observation
config = PickConfig(observations=[
    EndEffectorPose(),
    GraspState(),
    ObjectPose(),
    ObjectOffset(),
    WristCamera(width=224, height=224),
])

With obs_mode="state" (the default), a list with a camera component produces a dictionary. Its "state" key holds every selected state component. Each camera has a separate key, such as "wrist_camera" or "overhead_camera".

Default Observations by TaskCopy Anchor Link

Each task config auto-populates observations if you don't provide one:

PickAndPlace and StackCube carry ObjectVelocity because their success predicates require the object to have settled (object_static_lin_threshold / cube_static_lin_threshold), which no other component exposes. Success predicates per task are in Environments.

Observation ModesCopy Anchor Link

The obs_mode config parameter controls the semantic intent of the observation:

obs_mode="state" (default)Copy Anchor Link

The observation contains whatever components are listed in observations. This is useful for state-based reinforcement learning where the policy has access to ground-truth information.

obs_mode="visual"Copy Anchor Link

Designed for vision-based policies. Requires at least one camera component (e.g. WristCamera() or OverheadCamera()) in the observations list. Construction raises an error if no camera component is present.

The returned dictionary contains the camera keys and a "state" key with JointPositions only. info["privileged_state"] contains the full selected state for a critic or diagnostic code, not as policy input.

from so101_nexus import PickConfig, JointPositions, WristCamera

config = PickConfig(
    obs_mode="visual",
    observations=[JointPositions(), WristCamera(width=224, height=224)],
)

Units: Degrees vs RadiansCopy Anchor Link

Configuration APIs (REST_POSE, *_deg fields, spawn ranges, leader-arm helpers) use degrees per the library convention. Runtime values exposed by the env (obs["state"], env.step(action), env.action_space.low/high) use radians, which is the native unit of the underlying MuJoCo and MuJoCo Warp engines and the unit the integrator consumes when writing ctrl. Converting on every step would add runtime work without changing the simulation result.

Code that crosses into LeRobot territory converts at the boundary:

• The teleop recorder (so101_nexus.teleop) stores actions and states in degrees.
• The rollout recorder (so101_nexus.policy_adapters) reads obs["state"] and converts to degrees before calling select_action, then converts the policy's degree-valued action back to radians before env.step.

This matches LeRobot's use_degrees=true convention. For background on why LeRobot uses degrees, mid-range zero, and no wrap-around, see Backward compatibility.

Inspecting ObservationsCopy Anchor Link

import gymnasium as gym
import so101_nexus.mujoco

# State-only (default)
env = gym.make("MuJoCoPickLift-v1")
obs, info = env.reset()
print(f"Observation shape: {obs.shape}")  # (31,)
env.close()

# With camera
from so101_nexus import (
    PickConfig, EndEffectorPose, GraspState, ObjectPose, ObjectOffset, WristCamera,
)

config = PickConfig(observations=[
    EndEffectorPose(),
    GraspState(),
    ObjectPose(),
    ObjectOffset(),
    WristCamera(width=224, height=224),
])
env = gym.make("MuJoCoPickLift-v1", config=config)
obs, info = env.reset()
print(obs["state"].shape)          # (18,): state components
print(obs["wrist_camera"].shape)   # (224, 224, 3): camera image
env.close()

Choosing the Right SetupCopy Anchor Link

Specialist detailsCopy Anchor Link