Observations

Every SO101-Nexus environment builds its observation from a list of observation components. Components are lightweight descriptor classes that tell the environment which data to include. State components contribute slices of a flat vector; camera components add image tensors to a dictionary observation.

Observation Components

State Components

State components produce fixed-size slices of the observation vector.

GazeDirection and GazeState are anchored at the wrist camera, not at the gripper tip. Tasks without a target object (Move) do not support either component.

Camera Components

Camera components add image tensors to a dict-style observation space. They have size = 0 in the state vector.

Camera components accept width and height parameters (default 640x480). WristCamera also supports domain randomization parameters for FOV and pitch.

from so101_nexus import WristCamera, OverheadCamera

# Default resolution
wrist = WristCamera()

# Custom resolution with FOV randomization
wrist = WristCamera(width=224, height=224, fov_deg_range=(60.0, 90.0))

# Overhead camera
overhead = OverheadCamera(width=320, height=240, fov_deg=45.0)

Both backends render camera observations, and observation keys, shapes, and dtypes match. Pixel values do not: the MuJoCo backend rasterizes on CPU through per-env OpenGL renderers, while the MuJoCo Warp backend ray-traces all worlds in one batched pass on the simulation device, returning uint8 tensors of shape (num_envs, height, width, 3). Warp camera observations are independent of Gymnasium render_mode; render_mode visualization is MuJoCo-only. See Backends.

Composing Observations

Pass a list of components via the observations parameter on any config. Each task provides sensible defaults when observations is not specified.

from so101_nexus import (
    PickConfig, JointPositions, JointVelocities, EndEffectorPose, GraspState,
    ObjectPose, ObjectOffset, WristCamera,
)

# State-only observations
config = PickConfig(observations=[
    JointPositions(),
    JointVelocities(),
    EndEffectorPose(),
    GraspState(),
    ObjectPose(),
    ObjectOffset(),
])

# Add a wrist camera to the observation
config = PickConfig(observations=[
    EndEffectorPose(),
    GraspState(),
    ObjectPose(),
    ObjectOffset(),
    WristCamera(width=224, height=224),
])

When the observation list contains only state components, the observation is a flat NumPy array. When it contains one or more camera components, the observation becomes a dictionary with a "state" key (flat vector from all state components) plus one key per camera (e.g. "wrist_camera", "overhead_camera").

Default Observations by Task

Each task config auto-populates observations if you don't provide one:

PickAndPlace and StackCube carry ObjectVelocity because their success predicates require the object to have settled (object_static_lin_threshold / cube_static_lin_threshold), which no other component exposes. Success predicates per task are in Environments.

Observation Modes

The obs_mode config parameter controls the semantic intent of the observation:

obs_mode="state" (default)

The observation contains whatever components are listed in observations. This is useful for state-based reinforcement learning where the policy has access to ground-truth information.

obs_mode="visual"

Designed for vision-based policies. Requires at least one camera component (e.g. WristCamera() or OverheadCamera()) in the observations list. Construction raises an error if no camera component is present.

from so101_nexus import PickConfig, JointPositions, WristCamera

config = PickConfig(
    obs_mode="visual",
    observations=[JointPositions(), WristCamera(width=224, height=224)],
)

Units: Degrees vs Radians

Configuration APIs (REST_POSE, *_deg fields, spawn ranges, leader-arm helpers) use degrees per the library convention. Runtime values exposed by the env (obs["state"], env.step(action), env.action_space.low/high) use radians, which is the native unit of the underlying MuJoCo and MuJoCo Warp engines and the unit the integrator consumes when writing ctrl. Converting on every step would add runtime work without changing the simulation result.

Code that crosses into LeRobot territory converts at the boundary:

• The teleop recorder (so101_nexus.teleop) stores actions and states in degrees.
• The rollout recorder (so101_nexus.policy_adapters) reads obs["state"] and converts to degrees before calling select_action, then converts the policy's degree-valued action back to radians before env.step.

This matches LeRobot's use_degrees=true convention. For background on why LeRobot uses degrees, mid-range zero, and no wrap-around, see Backward compatibility.

Inspecting Observations

import gymnasium as gym
import so101_nexus.mujoco

# State-only (default)
env = gym.make("MuJoCoPickLift-v1")
obs, info = env.reset()
print(f"Observation shape: {obs.shape}")  # (31,)
env.close()

# With camera
from so101_nexus import (
    PickConfig, EndEffectorPose, GraspState, ObjectPose, ObjectOffset, WristCamera,
)

config = PickConfig(observations=[
    EndEffectorPose(),
    GraspState(),
    ObjectPose(),
    ObjectOffset(),
    WristCamera(width=224, height=224),
])
env = gym.make("MuJoCoPickLift-v1", config=config)
obs, info = env.reset()
print(obs["state"].shape)          # (18,): state components
print(obs["wrist_camera"].shape)   # (224, 224, 3): camera image
env.close()

Choosing the Right Setup

Specialist details