New NVIDIA Cosmos 3 Open Models for Interactive World Simulation

NVIDIA has significantly advanced its research capabilities by releasing four new open-weight checkpoints under its Cosmos 3 world model development platform. This suite of models is designed to generate highly interactive and complex simulated worlds, making it a powerful tool for researchers and developers working in fields like autonomous driving, advanced robotics, and general AI agent development. The core goal of the Cosmos 3 platform is to provide a foundational, adaptable starting point, allowing the community to build richer, more responsive simulated environments that mimic the complexities of the real world.

The release is particularly significant because it addresses the fundamental challenge of data scarcity and variability in real-world AI training. Instead of relying solely on expensive, limited, or difficult-to-label real-world data, these models allow users to synthetically generate vast amounts of diverse, multi-sensor data under controlled, yet highly variable, conditions. All four models are based on the Cosmos 3 Nano architecture and possess 16 billion trainable parameters, ensuring a high degree of complexity and potential for deep learning applications. Furthermore, the models utilize OpenMDW1.1, which permits both commercial and non-commercial use, greatly lowering the barrier to entry for industry adoption.

One of the flagship models is Cosmos3-Nano-Transfer-Auto. This checkpoint is a bidirectional system designed for generating offline synthetic driving data. Its capability to generate synchronized video across up to eleven different camera views, including specialized fisheye lenses, is groundbreaking. Crucially, it supports joint generation of both RGB (color) and LiDAR (depth/geometry) data. The system is controlled by Per-camera World Scenario Map (WSM) controls, which dictate the overall scene structure and the movement of traffic elements. Complementing this are text prompts, which guide the visual appearance of the scene, allowing users to specify details beyond just the geometry. Developers can manipulate environmental conditions—such as varying weather, lighting, and time of day—to massively expand the coverage of their training and evaluation datasets. The inclusion of optional initial RGB images helps anchor the scene's visual appearance, while the joint LiDAR generation further leverages aligned HD-map range-map controls, ensuring that the generated geometry aligns perfectly with the visual data. This comprehensive, multi-modal, and controllable data generation capability makes it an unparalleled tool for training robust autonomous systems.

Building upon the foundation of Transfer-Auto is Cosmos3-Nano-Sim-Auto. This model represents a crucial evolution: it is the causally distilled, streaming variant of Transfer-Auto. While Transfer-Auto is excellent for large, offline data batches, Sim-Auto is optimized for real-time, interactive simulation. It generates successive, small chunks of synchronized camera video and LiDAR data in just a few denoising steps. This speed and efficiency are achieved by using the generated history and updated driving controls to support fast streaming and long, interactive rollouts. For practical use, a developer first adapts Transfer-Auto to their specific sensor rig and domain (e.g., changing camera layout or target environment) and then distills it into Sim-Auto. This two-step process ensures that the model remains highly specialized while gaining the necessary streaming performance for closed-loop interaction. The process requires a simulation runtime and a WSM renderer to continuously update the controls based on the agent's driving inputs, enabling true, continuous interaction.

For the specialized field of robotics, the platform introduces Cosmos3-Nano-Sim-Bimanual. This checkpoint is designed to simulate how an entire environment evolves in response to the actions of a bimanual robot. It supports interactive two-arm simulation, which can be controlled either by a sophisticated AI policy or by human-operated joysticks. The process is highly dynamic: a user starts with an initial image, moves the robot or camera, and the model then generates the subsequent observations, effectively simulating the physical consequences of the robot's actions on the environment. This capability is vital for training robots to interact safely and effectively with complex, dynamic settings.

Another specialized model is Cosmos3-Nano-Sim-Transfer. This checkpoint bridges the gap between traditional physics simulators and visual observation generation. In a typical simulation pipeline, the simulator handles the physics, motion, and physical state updates. Sim-Transfer takes the simulator's spatial controls and a simple text prompt, and then renders the resulting color observations that an AI policy can consume. The workflow is clear: an action is taken, the simulator updates the physical state, the spatial controls are updated accordingly, and finally, the model generates the corresponding RGB image. This separation of concerns—where the simulator maintains ground truth physics and the model generates the visual appearance—is a powerful paradigm for research, allowing researchers to test policies against highly realistic, yet fully controllable, visual inputs.

Collectively, these checkpoints provide complementary ways to build sophisticated simulation environments. For autonomous driving, the emphasis on synchronized camera and LiDAR generation ensures that the visual appearance and the underlying 3D geometry remain perfectly consistent, which is non-negotiable for safety-critical systems. Furthermore, the ability to vary scene and traffic-signal controls allows researchers to systematically study how a driving policy reacts to rare or challenging edge cases, greatly improving the robustness of the final product. The platform's open weights and detailed post-training recipes are NVIDIA's commitment to democratizing world modeling, providing a shared starting point for the entire AI research community to compare methods, adapt models, and contribute improvements. While the work ahead is substantial—requiring rigorous measurement of where the models faithfully support policy learning, understanding their inherent limitations, and improving performance in challenging scenarios—the Cosmos 3 suite establishes a powerful, open foundation for the next generation of embodied AI and simulation research.