Constructing realistic 3D environments for robotics simulation has traditionally been a labor-intensive process, often requiring weeks of manual modeling and setup. Now, with generative world models, you may go from a text prompt to a photorealistic, simulation-ready world in a fraction of time. By combining NVIDIA Isaac Sim, an open source robotics reference framework, with generative models similar to Marble from World Labs , you can create entire 3D scenes for robotics development from a text or image prompt.
World Labs recently published the case study “Scaling Robotic Simulation with Marble,” showing how researchers are using Marble’s generative worlds to speed up robot training, testing, and sim-to-real transfer.
On this tutorial, we’ll walk through an end-to-end workflow:
- Scene export: Export an existing scene from Marble gallery as Gaussian splats (PLY) and a collider mesh (GLB)
- Scene conversion: Convert the Marble outputs to USD format using NVIDIA Omniverse NuRec
- Scene import and construction: Import into NVIDIA Isaac Sim
- Simulation in Isaac Sim: Add a robot and run the simulation.
By the tip, you’ll have a practical virtual environment where robots can interact physically, all generated way more quickly than by traditional methods. Let’s dive in.
Step 1: Get a 3D kitchen scene from World Labs Marble
World Labs Marble produces wealthy visual detail and geometric data like depth and surface normals, together with an exportable collider mesh for physical simulation.
For this tutorial, as an alternative of generating a brand new kitchen from scratch, we’ll use a pre-made Marble kitchen scene that’s available in Marble’s example gallery. This protects time and ensures we’ve a practical environment able to go. The chosen scene is an in depth kitchen and front room interior, complete with furniture and typical kitchen items.
Steps to export the kitchen world from Marble:
- Log in to Marble: Register to your Marble account on the internet. Once logged in, navigate to the pre-made kitchen scene.
- Open the scene: Click on the world to load it in Marble’s 3D viewer. You’ll be able to explore it with WASD controls and mouse as in case you were in a game, to confirm it looks good.
- Download the world: Find the Download button in the underside bar of Marble’s interface.
- Select “Splats (PLY)” to download a Gaussian splat representation. Marble’s Gaussian splat is provided as a .ply file, which accommodates hundreds of thousands of semi-transparent particles representing the scene with high fidelity.
- Select “Collider Mesh (GLB)” to download the triangle mesh of the scene. It will contain the geometry of the kitchen as an ordinary glTF model.
Note that exporting PLY and GLB files in World Labs Marble requires a paid plan. In case you don’t have one, World Labs provides sample PLY and GLB files from its gallery. For this tutorial, we’ll use the kitchen scene PLY and GLB files as our example. Save the files as MarbleKitchenwithLight.ply and MarbleKitchenwithLight_collider.glb.
At this point, we’ve our kitchen environment in two forms—as Gaussian splats and as a triangle mesh. Each serves a special purpose: The PLY captures the complete visual detail of the scene, and the GLB provides the mesh geometry needed for physics and collisions in simulation.
Step 2: Convert downloaded PLY into USDZ
NVIDIA Isaac Sim uses Universal Scene Description (USD) as its scene format. To make use of our Marble-generated world in Isaac Sim, we’d like to convert the exported PLY into USD format. We are going to then import it, profiting from NVIDIA Omniverse NuRec capabilities to render the point-based scene efficiently.
On the core of NuRec is the 3DGUT algorithm for Gaussian-based reconstruction and rendering. The NVIDIA 3DGRUT repository accommodates a script to convert a .ply splat file right into a USDZ file, a zip-compressed archive that accommodates USD-specific data. We are going to use this to convert our Marble PLY:
1. Arrange 3DGRUT: Clone the 3DGRUT repository and install its environment. On this tutorial, we arrange 3DGRUT inside a dedicated Conda environment named “3dgrut.”
The environment requires Linux with an NVIDIA GPU, CUDA 11.8+, and GCC 11 or lower. In case you have already got a Python environment with the needed libraries (PyTorch, etc.), you may alternatively just run the conversion Python script in that environment.
git clone --recursive https://github.com/nv-tlabs/3dgrut.git
cd 3dgrut
chmod +x install_env.sh
./install_env.sh 3dgrut
conda activate 3dgrut
2. Convert PLY to USDZ: Once 3DGRUT is ready up, use the provided conversion script to show the Marble point cloud into USDZ:
$ python -m threedgrut.export.scripts.ply_to_usd
/path/to/MarbleKitchenwithLight.ply
--output_file /path/to/MarbleKitchenwithLight.usdz
This command will read the .ply file and produce a .usdz file. USDZ uses a custom USD schema (an extension of UsdVolVolume) to represent the Gaussian splats in a way that Omniverse can render. Essentially, it embeds the purpose cloud as a volumetric primitive, preserving the visual fidelity of the Marble scene. For more details on NuRec neural volumes and the way they’re rendered in Omniverse, see the NuRec Rendering documentation.
Now, we’ve one USDZ file and one GLB file:
- MarbleKitchenwithLight.usdz – the visual splat world
- MarbleKitchenwithLight_collider.glb – the collider mesh we’ll use for physics.
Step 3: Import USDZ/GLB into Isaac Sim and construct the scene
After generating the USDZ file, the following step is to bring the kitchen scene into Isaac Sim, align the mesh with the Gaussian splats, and add physics and lighting so it is prepared for interaction.
Since we’re editing the scene contents, we’d like to extract the USDZ archive. Unzip the file and open the default.usda file generated, then undergo the next steps:
Geometrically align the Gaussian volume:
We wish to ensure that that the origin of the imported scene and its scale matches Isaac Sim. With a purpose to do this:
- Add a ground plane to the scene. This shall be used as a reference for the bottom of the imported Gaussian volume, and function a smooth collider.
- The imported Gaussian volume is contained in an “xform” primitive, which is used to rework the amount. To align the amount with the ground, select the xform primitive and adjust its “Translate” values so the ground of the kitchen sits exactly on the bottom plane. Use the bottom plane as a visible reference and move the Gaussian volume until the purpose cloud’s floor coincides with it.
- The generated scene could also be smaller or larger than the real-world scale. To roughly match the real-world scale, we will use a default cube as a visible reference, which has 1 meter side length. After inserting a cube object, we will adjust the general X, Y, and Z scaling accordingly. For our example kitchen scene, an element of two for the scaling gives roughly the proper sizing, e.g., for the cupboard and stove.
- Finally, fine-tune the rotation of the xform primitive to ensure that the Gaussian point cloud aligns with the bottom plane as accurately as possible. A straightforward approach to confirm that is to make use of the tiles on the kitchen wall as a reference and rotate the Gaussian such that they’re completely parallel to the bottom plane created. Once aligned, move the bottom plane back down so it sits exactly on the kitchen floor level.
Add physics and lighting to the scene:
Now that we’ve aligned the imported Gaussians, we would like so as to add physics and lighting in order that shadows and object interactions work as expected.
We are going to use the cube that we previously created to regulate the scene scale again to check shadows and physics.
- Within the collision mesh of the bottom plane, activate the matte object property. This ensures it really works properly as a shadow receiver.
- Add a dome light to the scene.
- Select the ‘gauss’ Volume prim within the stage window, then within the property window, scroll right down to “Raw USD Properties” and click on the triangle to disclose additional settings. Then, scroll to the “proxy” field, and click on on “Add Goal.” Finally, select the GroundPlane CollisionMesh because the goal.
Move the cube around to make sure shadows show up as expected.
On setting the cube as a rigid body with colliders and hitting play within the simulation, the cube interacts with the bottom plane as expected. Nonetheless, it “goes through” the Gaussians. Allow us to now move on to organising the physics of the Gaussian representation.
The collision information for the Gaussians is within the GLB file. Import this mesh, align it with the Gaussian volume, and enable it as a collider.
- Drag and drop the MarbleKitchenwithlight_collider.glb file under the Gaussian volume. Make certain it’s under the Gaussian volume, because the hierarchy is very important. The collider will show up within the scene.
- Zoom out of the scene a bit and set the X rotation to -90 to match the coordinate conventions of the Gaussian volume. Now the rendered volume and the collision mesh align completely.
- Enable the physics collider preset for the imported collision mesh.
- Turn off the visibility for the collider because it is overlaid with the Gaussian volume. This affects only the visuals of the scene; physics will use the colliders we just arrange within the scene.
The geometry, physics, and lighting for the scene are actually in fine condition: The Gaussian volume provides the photoreal visuals, while the GLB collider and ground plane handle physics and shadows. The scene is now ready for a robot to be added.
Step 4: Add a robot and run the simulation
With the kitchen scene aligned and physics enabled, the ultimate step is so as to add a robot and drive it around to validate the setup.
- Drag and drop the NVIDIA Nova Carter robot into the scene.
- Add a differential controller for the robot with and enable keyboard control. It will create the vital motion graph, which allows us to make use of our keyboard to maneuver the robot around.
- Change to a camera mounted on the robot and hit play. Move the robot around with WASD and confirm that it respects the kitchen geometry: It should rest on the ground, collide with counters and furniture, and never fall through the scene.
At this point, the Marble kitchen scene is fully integrated into Isaac Sim as a physics-enabled environment, and you may drive robots interactively through it.
Summary
On this tutorial, we downloaded an AI-generated 3D environment complete with geometry after which brought it into Isaac Sim as a simulation-ready scene. We arrange robots in an AI-generated world. The top-to-end workflow here can now be accomplished in mere hours. This ability to rapidly generate various high-fidelity worlds unlocks more scalable robot development in simulation. With Marble and Isaac Sim, in case you can describe a world, you’ll likely give you the option to begin testing it the identical day.
To learn more, try the next:
- Create your personal custom environment with World Labs Marble – You’ll be able to start with a text description, a single image, multiple photos from different angles, or perhaps a rough 3D layout.
- Create your personal custom environment with input image and use it for Isaac Sim with Lyra, an NVIDIA research initiative on generative 3D scene reconstruction via video diffusion model.
Learn more about simulation innovations and meet with NVIDIA experts at SIGGRAPH Asia, happening Dec. 15 to 18 on the Hong Kong Convention and Exhibition Centre.
