Constructing robust, intelligent robots requires testing them in complex environments. Nevertheless, gathering data within the physical world is dear, slow, and infrequently dangerous. It is almost unimaginable to soundly train for real-world critical risks, corresponding to high-speed collisions or hardware failures. Worse, real-world data is frequently biased toward “normal” conditions, leaving robots unprepared for the unexpected.
Simulation is important to bridge this gap, providing a risk-free environment for rigorous development. Nevertheless, traditional pipelines struggle to support the complex needs of contemporary robotics. Today’s generalist robots must master multimodal learning—fusing diverse inputs corresponding to vision, touch, and proprioception to navigate messy, unstructured worlds. This creates a brand new requirement for simulation: it must deliver scale, realism, and multimodal sensing multi functional tight training loop, something traditional CPU-bound simulators cannot handle efficiently.
This edition of NVIDIA Robotics Research and Development Digest (R²D²) explains how NVIDIA Isaac Lab, an open source GPU-native simulation framework from NVIDIA Research, unifies these capabilities in a single stack designed for large-scale, multimodal robot learning.
Key robot learning challenges
Modern robot learning in simulation pushes simulation infrastructure to its limits. To coach robust policies efficiently, researchers must overcome critical hurdles, including:
- Scaling simulation to hundreds of parallel environments to beat the slow training times of CPU-bound tools
- Integrating multiple sensor modalities (vision, force, and proprioception) into synchronized, high-fidelity data streams
- Modeling realistic actuators and control frequencies to capture the nuances of physical hardware
- Bridging the gap between simulation and real-world deployment through robust domain randomization and accurate physics
Isaac Lab: Open source, unified framework for robot learning
Isaac Lab is a GPU-accelerated simulation framework for multimodal robot learning. It’s a unified, GPU-native platform designed to resolve the challenges of contemporary robot learning. By consolidating physics, rendering, sensing, and learning right into a single stack, it provides researchers with the technology to coach generalist agents with unprecedented scale and fidelity.
Isaac Lab core elements
The important thing elements of Isaac Lab include:
- GPU-native architecture: Delivers end-to-end GPU acceleration for physics and rendering, enabling massive parallelism to drastically reduce training time.
- Modular and composable design: Features flexible components for diverse embodiments (humanoids, manipulators) and reusable environments to speed up development.
- Multimodal simulation: Leverages tiled RTX rendering and Warp-based sensors to generate wealthy, synchronized observations (vision, depth, tactile) alongside realistic multi-frequency control loops.
- Integrated workflows: Provides built-in support for reinforcement learning (RL) and imitation learning (IL), streamlining large-scale data collection, domain randomization, and policy evaluation. It connects out-of-the-box with top RL libraries including SKRL, RSL-RL, RL-Games, SB3, and Ray, and seamlessly integrates with NVIDIA Cosmos-generated data for augmented imitation learning.
Contained in the Isaac Lab framework: A modular toolkit
Isaac Lab breaks down robot learning into composable constructing blocks, enabling you to construct complex, scalable tasks without “reinventing the wheel.”
Features include a manager-based workflow, procedural scene generation, and more.
Manager-based workflow
As a substitute of writing monolithic scripts that blend physics and logic, Isaac Lab decouples your environment into separate “Managers” for observations, actions, rewards, and events. This makes your code modular and reusable. For instance, you may swap a robot’s reward function without touching its sensor setup.
@configclass
class MyRewardsCfg:
# Define rewards as weighted terms
track_lin_vel = RewTerm(func=mdp.track_lin_vel_xy_exp, weight=1.0, params={"std": 0.5})
penalty_lin_vel_z = RewTerm(func=mdp.lin_vel_z_l2, weight=-2.0)
@configclass
class MyEnvCfg(ManagerBasedRLEnvCfg):
# Plug within the reward config cleanly
rewards: MyRewardsCfg = MyRewardsCfg()
# ... other managers for actions, observations, etc.
Procedural scene generation
To forestall overfitting, you rarely need to train on a single static scene. With the Isaac Lab scene generation tools, you may define rules to spawn diverse environments procedurally. Whether it’s scattering debris for a navigation task or generating rough terrain for locomotion, you define the logic once, and the framework builds hundreds of variations on the GPU.
# Configure a terrain generator with diverse sub-terrains
terrain_cfg = TerrainGeneratorCfg(
sub_terrains={
"pyramid_stairs": MeshPyramidStairsTerrainCfg(
proportion=0.2, step_height_range=(0.05, 0.2)
),
"rough_ground": MeshRandomGridTerrainCfg(
proportion=0.8, noise_scale=0.1
),
}
)
More features
As well as, Isaac Lab provides:
- A unified asset API for importing any robot from USD, URDF, or MJCF
- Realistic Actuators to model motor dynamics, alongside 10+ Sensor types starting from IMUs to photorealistic RTX cameras
- A built-in teleoperation stack to further simplify data collection
Together, these features provide what you have to efficiently move from prototype to deployed policy.
Delivering GPU-accelerated performance at scale
Isaac Lab delivers the large throughput required for contemporary robot learning, achieving 135,000 FPS for humanoid locomotion (Unitree H1) and over 150,000 FPS for manipulation (Franka Cabinet)—training policies in minutes somewhat than days. Its unified GPU architecture eliminates CPU bottlenecks, maintaining high throughput even with complex RGB-D sensors enabled across 4,096 environments.
Benchmarks confirm linear scaling with VRAM and successful zero-shot transfer for diverse embodiments, including dexterous hands, multi-agent swarms, and the H1 humanoid walking robustly outdoors.
A canonical robot learning workflow
Isaac Lab standardizes the robot learning loop into a transparent, Python-first workflow. Whether you’re training a locomotion policy or a manipulation skill, the method follows the identical 4 steps: design, randomize, train, and validate.
To run an entire example—training a humanoid to walk—right out of the box, follow the steps below.
Step 1: Design and configure
First, define your environment in Python. Select your robot (Unitree H1, for instance), sensors, and randomization logic using a configuration class:
# pseudo-code representation of a config
@configclass
class H1FlatEnvCfg(ManagerBasedRLEnvCfg):
scene = InteractiveSceneCfg(num_envs=4096, env_spacing=2.5)
robot = ArticulationCfg(prim_path="{ENV_REGEX_NS}/Robot", spawn=...)
# Randomization and rewards are defined here
For more details, see the H1 Humanoid Environment Configuration within the isaac-sim/IsaacLab GitHub repo. Optionally, you may include additional sensors. Configuring your sensors is simple.
Configure a tiled camera:
from isaaclab.sensors import TiledCameraCfg
# Define a camera attached to the robot's head
tiled_camera: TiledCameraCfg = TiledCameraCfg(
prim_path="{ENV_REGEX_NS}/Robot/head/camera",
offset=TiledCameraCfg.OffsetCfg(
pos=(-7.0, 0.0, 3.0),
rot=(0.9945, 0.0, 0.1045, 0.0),
convention="world"),
data_types=["rgb"],
spawn=sim_utils.PinholeCameraCfg(
focal_length=24.0,
focus_distance=400.0,
horizontal_aperture=20.955,
clipping_range=(0.1, 20.0)
),
width=80,
height=80,
)
Configure a ray-caster (LiDAR):
from isaaclab.sensors import RayCasterCfg, patterns
# Define a 2D LiDAR scanner
lidar = RayCasterCfg(
prim_path="{ENV_REGEX_NS}/Robot/base_link/lidar",
update_period=0.1, # Run at 10Hz
offset=RayCasterCfg.OffsetCfg(pos=(0.0, 0.0, 0.2)),
attach_yaw_only=True, # Stabilize against robot tilt
pattern_cfg=patterns.LidarPatternCfg(
channels=32,
vertical_fov_range=(-15.0, 15.0),
horizontal_fov_range=(-180.0, 180.0)
)
)
Step 2: Train the policy
Next, launch a training script to start out learning. Isaac Lab uses the gymnasium interface, so it connects easily to RL libraries like RSL-RL or SKRL.
# Train a policy for the Unitree H1 humanoid
# This runs 4096 environments in parallel in your GPU
python source/standalone/workflows/rsl_rl/train.py --task=Isaac-Velocity-Flat-H1-v0
Step 3: Play and visualize
Once training is complete, confirm the policy by running it in inference mode. This loads the trained checkpoint and renders the result.
# Run the trained policy and visualize the robot walking
python source/standalone/workflows/rsl_rl/play.py --task=Isaac-Velocity-Flat-H1-v0
Step 4: Sim-to-real deployment
After validation, the policy may be exported to ONNX or TorchScript for deployment on physical hardware, leveraging the domain randomization applied during training. To see real-world examples, see the Sim-to-Real Deployment Guide.
Ecosystem adoption
Leading organizations and research labs in humanoid robotics, embodied AI, and legged locomotion are deploying Isaac Lab to speed up the event of generalist robot policies and foundation models, including:
- Agility Robotics’ general-purpose humanoid, Digit, uses the Isaac Lab framework to refine whole-body control through tens of millions of reinforcement learning scenarios, which speed up enhancements to its skill sets corresponding to step recovery from environmental disturbances, often needed in highly dynamic areas like manufacturing and logistics facilities.
- Skild AI is constructing a general-purpose robotics foundation model that spans legged, wheeled and humanoid robots, using Isaac Lab for locomotion and dexterous manipulation tasks training and NVIDIA Cosmos world foundation models for generating training datasets.
- FieldAI is training cross-embodied robot brains for monitoring and inspection in construction, manufacturing, and oil and gas environments, using Isaac Lab for reinforcement learning and NVIDIA Isaac Sim for synthetic data generation and software-in-the-loop validation.
- The Robotics and AI Institute uses NVIDIA Isaac Lab to coach high-performance reinforcement learning controllers for agile legged locomotion, dynamic whole-body manipulation, and custom robotics platforms, optimizing simulator parameters to shut the sim-to-real gap before deploying policies on Boston Dynamics Spot and Atlas, and RAI’s Ultra Mobile Vehicle (UMV).
- UCR is constructing rugged humanoid robots for heavy industries on the NVIDIA Isaac platform, using Isaac GR00T’s synthetic data pipelines, Isaac Lab, and Isaac Sim to coach end‑to‑end mobility policies and iteratively close sim-to-real gaps for robust deployment of Moby in harsh construction and industrial sites.
Start with multimodal robot learning
Able to scale your personal multimodal robot learning workloads with Isaac Lab? Start here with core resources and level up with the newest research for advanced workflows.
Learn more about how researchers are leveraging simulation and generative AI to push the boundaries of robot learning:
- Harmon: Combines language models and physics to generate expressive whole-body humanoid motions directly from text.
- MaskedMimic: A generalist control policy that learns diverse skills through motion inpainting, simplifying humanoid control without complex rewards.
- SIMPLER: A framework for evaluating real-world manipulation policies (RT-1, Octo) in simulation to reliably predict physical performance.
NVIDIA GTC AI Conference is occurring March 16–19, 2026 in San Jose with a must-see keynote with CEO Jensen Huang at SAP Center on March 16 at 11:00 a.m., Pacific time. Discover GTC robotics sessions on how AI, simulation, and accelerated computing are enabling robots to see, learn, and make decisions in real time.
This post is a component of our NVIDIA Robotics Research and Development Digest (R2D2) series that helps developers gain deeper insight into the SOTA breakthroughs from NVIDIA Research across physical AI and robotics applications.
Not sleep-to-date by subscribing to the newsletter and following NVIDIA Robotics on YouTube, Discord, and developer forums.
To start in your robotics journey, enroll in free NVIDIA Robotics Fundamentals courses.
