Backends
Choosing between MuJoCo and MuJoCo Warp, and the vectorization, rendering, and physics differences between them.
SO101-Nexus uses MuJoCo as its default simulation backend. A MuJoCo Warp backend adds native GPU-parallel batching for reinforcement learning. Both cover the same SO-101 tasks, with the same env IDs modulo the MuJoCo/Warp prefix. See Environments for the ID list.
The MuJoCo backend is stable. The MuJoCo Warp backend is experimental: its API and physics may change between minor releases. See Stability and versioning.
Vectorization
MuJoCo environments are vectorized through Gymnasium's standard gym.vector.SyncVectorEnv wrapper, which runs multiple environment instances on CPU.
The MuJoCo Warp backend instead batches simulation natively on the GPU. Construct any task with gym.make_vec("WarpPickLift-v1", num_envs=N, device="cuda", vectorization_mode="vector_entry_point"); reset and step return torch CUDA tensors of shape (num_envs, ...).
Warp envs route the same composable state observation components as MuJoCo (see the component table for the full list, which the two backends share). They also support WristCamera and OverheadCamera observation components, rendered in one batched ray-tracing pass on the simulation device. Cameras support RGB and depth, with per-world wrist-camera domain randomization. See Depth observations for the output contract.
Render Modes
MuJoCo envs support Gymnasium render_mode="human", render_mode="rgb_array", and render_mode="depth_array".
The depth render mode uses the same metric depth contract as camera observations.
The rendered view is selected by RenderConfig.
Warp envs support render_mode="rgb_array" and render_mode="depth_array" through gym.make_vec(). Call env.render() to get a torch tensor on the simulation device: RGB has shape (num_envs, height, width, 3) and dtype uint8; metric depth has shape (num_envs, height, width) and dtype float32. Each call returns independent storage. Warp does not support human rendering and rejects unsupported modes.
Visualization renders only when requested, separately from policy camera observations. The render context is allocated on the first call. With render_mode=None, Warp adds no visualization camera or render context. Use RenderConfig to select a view and optional placement ranges.
Object Pools and the Batched Model
WarpPickLift and WarpTouch accept the same heterogeneous object pools as MuJoCo (CubeObject, YCBObject, MeshObject, plus n_distractors), WarpPickAndPlace accepts a carried-object pool plus one slot per distractors pool entry when n_distractors > 0, and WarpStackCube compiles one cube slot per configured color in cube_a_colors/cube_b_colors plus the same distractor slots. Each pool object is compiled as a freejoint slot, the per-world target (or, for pick-and-place and stack-cube, the role slots plus the sampled distractors) is selected at reset, and inactive slots are parked off-world for physics. A separate rendering view hides inactive slots below the floor in each world, excluding them from RGB, depth, and shadows while keeping active distractors visible. A pick-and-place target is only ever drawn from the carried pool, so reset(options={"target_index": k}) indexes that pool alone.
Because all worlds share one compiled model, per-episode geom_rgba color randomization of a single object is unsupported. Distinct colored cube slots still give per-world color variation through selection (this is how WarpStackCube matches the MuJoCo backend's per-episode resampling of both cube colors), and the WarpPickAndPlace goal disc color is fixed to the first configured color.
Vectorized worlds may carry different tasks, so env.task_descriptions holds one string per world and info["task_description"] is returned from reset() and step(). env.task_description reduces to the shared string, or a generic family string when worlds differ.
Physics Divergence
The historical CPU preset uses implicitfast and three NoSlip iterations. The Warp preset uses implicit and zero NoSlip iterations.
MuJoCo Warp 3.13 supports both integrators but does not implement NoSlip. Its device conversion also raises the native solver tolerance to 1e-6.
PhysicsConfig selects supported physics configuration on both backends and preserves command timing when the timestep changes. Matching configuration does not guarantee identical contacts or trajectories. The backends use different floating-point arithmetic and collision algorithms. The optional Warp box collision patch addresses the measured box contact mismatch in version 3.13.0. Validate transferred policies on the target backend.
Installation and Hardware
Install the backend with the warp extra, which needs an NVIDIA GPU and CUDA >= 12.8 for GPU execution. Warp has no ROCm/AMD support. To train on an AMD GPU you install the rocm extra instead, which affects only the MuJoCo backend's training loop; see Installation.
Objects
Both backends support CubeObject, YCBObject, and MeshObject, configured through the same object-pool fields (objects, n_distractors, and distractors on the pick-and-place and stack-cube families). See Customization for the recipes and Objects for the constructor reference.