Construct Synthetic Data Pipelines to Train Smarter Robots with NVIDIA Isaac Sim

MobilityGen is a workflow for generating data for mobile robots built on Isaac Sim. It supports data collection through manual methods like keyboard and gamepad teleoperation, and thru automated methods like random accelerations and random path following. 

In the next example, you’ll find out how MobilityGen is used to generate data for an H1 humanoid robot in Isaac Sim. This workflow will be used for other robot embodiments, like quadrupeds and autonomous mobile robots (AMRs), and has been tested on the Spot and Carter robots.

While data from MobilityGen can train mobility policies for robots, performance improves when the info includes visual diversity. We’ll study augmenting data with visual diversity using NVIDIA Cosmos in the following section. 

The next outlines the steps involved in generating data using MobilityGen.

1. Construct an Occupancy Map: It is a grid-based representation of the robot’s environment where each cell represents the probability of being occupied by an obstacle.
2. Record a trajectory: A trajectory of a mobile robot specifies position, velocity, and orientation at every fast because it moves through its environment. 
3. Replay and render: You possibly can replay the generated trajectories to judge and visualize data.

The next videos show methods to generate synthetic data in Isaac Sim using MobilityGen.

In the next example, we use a warehouse environment available in Isaac Sim to run MobilityGen. You possibly can create your personal environment using the SimReady assets we learned about within the last section.

Steps for constructing an Occupancy Map

1. Load the warehouse stage:
   1. Open Content Browser (Window > Browsers > Content).
   2. Load the warehouse USD file in Isaac Sim/Environments/Simple_Warehouse/warehouse_multiple_shelves.usd.
2. Create the Occupancy Map
   1. Select Tools > Robotics > Occupancy Map to open the extension.
   2. Within the Occupancy Map window, set Origin to:
      1. X: 2.0
      2. Y: 0.0
      3. Z: 0.0
      4. Note: To input a price within the text box, Ctrl + left click to activate the input mode.
   3. Within the Occupancy Map window, set Upper Certain to:
      1. X: 10.0
      2. Y: 20.0
      3. Z: 2.0 (Assumes the robot can move under 2-meter overpasses)
   4. Within the Occupancy Map window, set Lower Certain to:
      1. X: -14.0
      2. Y: -18.0
      3. Z: 0.1 (Assume the robot can move over 5cm bumps)
   5. Click Calculate to generate the occupancy map.
   6. Click Visualize Image to view the occupancy map.
   7. Within the Visualization window, under Rotate Image, select 180.
   8. Within the Visualization window, under Coordinate Type, select ROS Occupancy Map Parameters File YAML
   9. Click Regenerate Image.
   10. Copy the YAML text generated to your clipboard.
   11. In a text editor of alternative, create a brand new file named ~/MobilityGenData/maps/warehouse_multiple_shelves/map.yaml.
       Please note: On windows, replace ~ with a directory of alternative.
   12. Paste the YAML text copied from the Visualization window into the created file.
   13. Edit the road image: warehouse_multiple_shelves.png to read image: map.png.
   14. Save the file.
   15. Back within the Visualization window, click Save Image.
   16. Within the tree explorer, open the folder ~/MobilityGenData/maps/warehouse_multiple_shelves.
   17. Under the file name, enter map.png.
   18. Click save.

Confirm that you simply now have a folder named ~/MobilityGenData/maps/warehouse_multiple_shelves/ with a file named map.yaml and map.png inside.

Steps for recording a trajectory

After making a map of the environment, you possibly can generate data with MobilityGen:

1. Enable the MobilityGen UI extension.
   1. Navigate to Window > Extensions and seek for MobilityGen UI.
   2. Click the toggle switch for the MobilityGen UI extension.
   3. Note: It’s best to see two windows appear. One window is the MobilityGen UI, the opposite is to display the Occupancy Map and visualizations. One window is perhaps hiding behind the opposite after they first appear, so we recommend dragging them right into a window pane to view each at the identical time.
2. Construct the scenario:
   1. Within the MobilityGen window under Stage, paste the next USD:
      http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/5.0/Isaac/Environments/Simple_Warehouse/warehouse_multiple_shelves.usd
   2. Within the MobilityGen window, under Occupancy Map, enter the trail to the map.yaml file created previously at
      ~/MobilityGenData/maps/warehouse_multiple_shelves/map.yaml
   3. Under the Robot dropdown, select H1Robot.
   4. Under the Scenario dropdown, select KeyboardTeleoperationScenario.
   5. Click Construct.
      After just a few seconds, confirm that you simply see the scene and that the occupancy map appears.
3. Test drive the robot using the next keys:
   1. W – Move forward
   2. A – Turn left
   3. S – Move backwards
   4. D – Turn right
4. Start recording:
   1. Click Start recording to start out recording a log.
   2. Move the robot around.
   3. Click Stop recording to stop recording.

The information is now recorded to ~/MobilityGenData/recordings by default.

Steps for replaying and rendering

After recording a trajectory, which incorporates data like robot poses, you possibly can now replay the scenario.

1. Use the replay_directory.py Python script that ships with Isaac Sim. To run the script, call the next from contained in the Isaac Sim directory:

./python.sh standalone_examples/replicator/mobility_gen/replay_directory.py --render_interval 40 --enable isaacsim.replicator.mobility_gen.examples

After the script finishes, confirm that you have got a folder ~/MobilityGenData/replays, which incorporates the rendered sensor data. You possibly can open this folder to explore the info.

There are examples on methods to load and work with the recorded data within the open source MobilityGen GitHub Repository. We recommend visualizing your recorded data by running the Gradio Visualization Script.

Find more information, resembling adding a custom robot, within the tutorial on Data Generation with MobilityGen.

Augment generated training data using NVIDIA Cosmos

After generating data using MobilityGen, use Cosmos Transfer to generate photorealistic videos from synthetic robot data. This adds visual variation to reduce the sim-to-real gap and improves policy performance after deployment.

Cosmos Transfer is a WFM that generates photorealistic videos from inputs of multiple video modalities like RGB, depth, and segmentation. Together with the input video, you possibly can give a text prompt with details guiding how you would like the generated video to appear like. The next is an example prompt:

A sensible warehouse environment with consistent lighting, perspective, and camera motion. Preserve the unique structure, object positions, and layout from the input video. Make sure the output exactly matches the segmentation video frame-by-frame in timing and content. Camera movement must follow the unique path precisely.

Video 4 shows methods to run Cosmos Transfer on MobilityGen data so as to add visual variation.

Video 5. The inference process to generate photorealistic videos

As seen within the video, an example command to run inference with Cosmos Transfer is:

export CUDA_VISIBLE_DEVICES="${CUDA_VISIBLE_DEVICES:=0,1,2,3}"
export CHECKPOINT_DIR="${CHECKPOINT_DIR:=./checkpoints}"
export NUM_GPU="${NUM_GPU:=4}"
PYTHONPATH=$(pwd) torchrun --nproc_per_node=$NUM_GPU --nnodes=1 --node_rank=0 cosmos_transfer1/diffusion/inference/transfer.py 
    --checkpoint_dir $CHECKPOINT_DIR 
    --video_save_folder outputs/example1_single_control_edge 
    --controlnet_specs assets/inference_cosmos_transfer1_single_control_edge.json 
    --offload_text_encoder_model 
    --offload_guardrail_models 
    --num_gpus $NUM_GPU

During closed-loop evaluation within the lab, a policy trained on synthetic and Cosmos-augmented data consistently outperformed a policy trained on synthetic data alone. The next scenarios are handled well by the policy trained on synthetic and Cosmos-augmented data:

• Navigating around transparent obstacles.
• Avoiding obstacles that mix into the background, like a gray pole on a gray floor.
• Going closer to obstacles and reducing the general distance traveled to get to a goal position.
• Navigating in dimly lit environments.
• Navigating narrow passages.  

You possibly can run Cosmos Transfer on any real or synthetic video data. One other example is the tutorial on Using Cosmos for Synthetic Dataset Augmentation. This tutorial explains methods to generate synthetic data using Replicator in Isaac Sim. 

Getting began

NVIDIA provides a comprehensive collection of OpenUSD resources to speed up your learning journey. Start with the self-paced Learn OpenUSD, Digital Twins, and Robotics training curricula that construct the foundational skills covered on this guide.

For professionals able to take the following steps of their robotics profession, the OpenUSD Development certification offers a professional-level exam that validates your expertise in constructing, maintaining, and optimizing 3D content pipelines using OpenUSD. Get OpenUSD certified in person at NVIDIA GTC Washington, D.C., and learn more about synthetic data for robot development in the course of the Physical AI and Robotics Day

Tune in to approaching OpenUSD Insiders livestreams and connect with the NVIDIA Developer Community. Not sleep up to now by following NVIDIA Omniverse on Instagram, LinkedIn,  X, Threads, and YouTube.