Photo-realistic 3D models of the Mcity Test Facility created using NVIDIA Omniverse NuRec

Mcity at the University of Michigan is using NVIDIA Omniverse NuRec to create a new digital twin of its Mcity Test Facility, enabling reconstruction of real-life driving scenarios as high-fidelity 3D renderings.

NuRec (Neural Reconstruction) is a suite of tools powered by artificial intelligence (AI) that transform images captured from multiple sensors, such as cameras and lidar, into a highly realistic 3D model. With NuRec, developers can create interactive, simulated environments from images of the real world to enable testing of AI agents for autonomous driving and other applications.

This is possible because NuRec incorporates a technique known as Gaussian Splatting to train millions of tiny, colored, and translucent 3D “splats” to match the captured images. This results in much more photorealistic 3D models than traditional mesh models, which use game engine technology for rendering.

“Gaussian Splatting is the newcomer that has burst onto the scene in the last handful of years and it has changed the game,” said Mcity Research Director Greg Stevens. “Now simulation can be used to develop and verify end-to-end AV stacks, and since it is simulation, not physical testing, this unlocks the ability to test billions of simulated miles, which is not feasible in the real world.”

The NuRec-enhanced model represents the next phase of NVIDIA-powered simulation at Mcity. Last year, Mcity integrated the NVIDIA Omniverse Blueprint for Autonomous Vehicle Simulation into its digital twin to enable physics-based modeling of camera, lidar, radar and ultrasonic sensor data.

The goal was to minimize the sometimes not so subtle differences between a simulated environment and its real-life counterpart, known as the sim-to-real gap. These variances have in the past been cited by engineers as a reason simulation is not robust enough for reliable AV testing. NuRec will minimize the sim-to-real gap even further by enabling the Mcity NuRec digital twin to be directly used to develop and test AI perception models.

Mcity is working with SaferDrive AI, a University of Michigan startup, to use NuRec to create the Mcity digital twin. SaferDrive AI will use this digital twin together with their simulation platform, TeraSim, which is an open-source generative AI tool that is realistic, automated, and scalable. TeraSim can add adversity, like vehicles, pedestrians or bad weather, into the driving scenes to accelerate the simulation testing.

Once the NuRec digital twin of Mcity is complete, it will be available open-source to anyone to use under an MIT License.

“Mcity is committed to providing easy access to open-source resources like the NuRec digital twin of our test track,” said Mcity Research Director Greg Stevens. “Giving researchers and software developers the tools they need to advance technology is part of our mission to spur innovation in mobility and transportation.”