SO101-Nexus
Concepts

Backends

Choosing between MuJoCo and MuJoCo Warp, and the vectorization, rendering, and physics differences between them.

SO101-Nexus uses MuJoCo for scalar simulation and native CPU batching. The optional MuJoCo Warp backend adds GPU batching. Both backends cover the same SO-101 tasks, with the MuJoCo and Warp ID prefixes. See Environments for the ID list.

The MuJoCo backend is stable. Native CPU batching and MuJoCo Warp are experimental and can change between minor releases. See Stability and versioning.

Vectorization

gym.make constructs one MuJoCo environment. gym.make_vec uses MuJoCo's native CPU batch API by default. For Gymnasium's standard wrappers, pass vectorization_mode="sync" or "async" explicitly.

The MuJoCo Warp backend instead batches simulation natively on the GPU. Construct any task with gym.make_vec("WarpPickLift-v1", num_envs=N, device="cuda", vectorization_mode="vector_entry_point"); reset and step return torch CUDA tensors of shape (num_envs, ...).

Warp envs route the same composable state observation components as MuJoCo (see the component table for the full list, which the two backends share). They also support WristCamera and OverheadCamera observation components, rendered in one batched ray-tracing pass on the simulation device. Cameras support RGB and depth, with per-world wrist-camera domain randomization. See Depth observations for the output contract.

Native CPU batching

The default installation includes native CPU batching. Construct a vector environment with an existing MuJoCo ID:

import gymnasium as gym
import so101_nexus.mujoco
from so101_nexus import TouchConfig

env = gym.make_vec(
    "MuJoCoTouch-v1",
    num_envs=16,
    num_threads=4,
    config=TouchConfig(),
)
obs, info = env.reset(seed=42)
env.action_space.seed(42)
obs, rewards, terminated, truncated, info = env.step(env.action_space.sample())
env.close()

num_threads=0 uses all logical CPUs. Reset and step return NumPy observations and arrays of rewards, terminations, and truncations. The same IDs support scalar environments through gym.make. The default autoreset mode is Gymnasium's NEXT_STEP. Pass autoreset_mode="SameStep" for immediate resets with final_obs and final_info, or "Disabled" for explicit resets. max_episode_steps overrides the registered episode limit; a negative value disables it. The LeRobot EnvHub factory also accepts these IDs and uses same-step autoreset.

Each world uses an independent copy of the task configuration and model parameters, including object selection, colors, wrist cameras, controls, and observation components. env.task_descriptions returns the current instruction for each world. Native CPU batching uses the MuJoCo physics preset and honors PhysicsConfig.

mujoco.batch.Batch steps all but the final physics substep on its native thread pool. The final substep runs on each task's MuJoCo data to preserve contact-force queries and observation timing. Integration state and mutable model fields are synchronized each step. Resets, rewards, observation assembly, inverse kinematics, and rendering run per world in Python. PickAndPlace v2 uses scalar physics for every substep because its support evaluator needs continuous contact checks. One-substep control intervals also use scalar physics. Throughput depends on the task and batch size; this adapter does not claim a measured speedup.

Camera components support the same RGB and metric depth observations as CPU MuJoCo. env.render() returns a tuple of per-world images using MuJoCo's rendering modes, including human when a display is available. Rendering is per world and requires a working OpenGL context. It is separate from the physics thread pool.

Render Modes

MuJoCo envs support Gymnasium render_mode="human", render_mode="rgb_array", and render_mode="depth_array". The depth render mode uses the same metric depth contract as camera observations. The rendered view is selected by RenderConfig. Warp envs support render_mode="rgb_array" and render_mode="depth_array" through gym.make_vec(). Call env.render() to get a torch tensor on the simulation device: RGB has shape (num_envs, height, width, 3) and dtype uint8; metric depth has shape (num_envs, height, width) and dtype float32. Each call returns independent storage. Warp does not support human rendering and rejects unsupported modes.

Visualization renders only when requested, separately from policy camera observations. The render context is allocated on the first call. With render_mode=None, Warp adds no visualization camera or render context. Use RenderConfig to select a view and optional placement ranges.

Object Pools and the Batched Model

WarpPickLift and WarpTouch accept the same heterogeneous object pools as MuJoCo (CubeObject, YCBObject, MeshObject, plus n_distractors), WarpPickAndPlace accepts a carried-object pool plus one slot per distractors pool entry when n_distractors > 0, and WarpStackCube compiles one cube slot per configured color in cube_a_colors/cube_b_colors plus the same distractor slots. Each pool object is compiled as a freejoint slot, the per-world target (or, for pick-and-place and stack-cube, the role slots plus the sampled distractors) is selected at reset, and inactive slots are parked off-world for physics. A separate rendering view hides inactive slots below the floor in each world, excluding them from RGB, depth, and shadows while keeping active distractors visible. A pick-and-place target is only ever drawn from the carried pool, so reset(options={"target_index": k}) indexes that pool alone.

Because all worlds share one compiled model, per-episode geom_rgba color randomization of a single object is unsupported. Distinct colored cube slots still give per-world color variation through selection (this is how WarpStackCube matches the MuJoCo backend's per-episode resampling of both cube colors), and the WarpPickAndPlace goal disc color is fixed to the first configured color.

Vectorized worlds may carry different tasks, so env.task_descriptions holds one string per world and info["task_description"] is returned from reset() and step(). env.task_description reduces to the shared string, or a generic family string when worlds differ.

Physics Divergence

Both backends share the PhysicsConfig defaults and jaw collision geometry. The PhysicsConfig reference defines the defaults and the grasp retention tests. An explicit configuration preserves command timing when the physics timestep changes. MuJoCo Warp 3.13 supports implicit and implicitfast but does not implement NoSlip. Warp rejects nonzero noslip_iterations and solver tolerances smaller than the float32 representation of 1e-6. Matching configuration does not guarantee identical contacts or trajectories. The backends use different floating-point arithmetic and collision algorithms. Warp selects its analytic box-box manifold before model conversion with MuJoCo's mjDSBL_NATIVECCD flag. The native convex path can lose support contacts between nearly aligned boxes after small changes in model inertia. This flag changes box-box detection in Warp. Mesh collisions retain the convex path. The optional Warp box collision patch addresses the measured box contact mismatch in version 3.13.0. Validate transferred policies on the target backend.

Installation and Hardware

Install the backend with the warp extra, which needs an NVIDIA GPU and CUDA >= 12.8 for GPU execution. Warp has no ROCm/AMD support. To train on an AMD GPU you install the rocm extra instead, which affects only the MuJoCo backend's training loop; see Installation.

Objects

Both backends support CubeObject, YCBObject, and MeshObject, configured through the same object-pool fields (objects, n_distractors, and distractors on the pick-and-place and stack-cube families). See Customization for the recipes and Objects for the constructor reference.

On this page