NVIDIA Brings Real-Time World Modeling to Edge Robots

NVIDIA has released Cosmos 3 Edge through the Cosmos 3 repository on Hugging Face. The open model has 4 billion parameters and is designed for machines operating outside data centers, including robots and vision-based AI systems in factories, warehouses, hospitals, and other physical environments. Its central goal is to provide strong perception, reasoning, prediction, and action-generation capabilities on devices with limited memory. NVIDIA presents Cosmos 3 Edge as a compact model that brings world understanding and robot control directly onto edge hardware, reducing the need to send every decision to a remote server.

The model can operate as a small vision-language model, interpreting visual observations together with text instructions while supporting real-time inference. NVIDIA says it delivers memory-efficient, high-throughput performance across its edge computing portfolio, including RTX PRO and GeForce RTX GPUs, DGX systems, and Jetson devices such as the newly announced Jetson T2000 and T3000 modules. As a post-trained world action model, it processes robot observations at a resolution of 640×360 pixels. On Jetson Thor, it can generate 32 actions per inference and support real-time robot control at 15 Hz, meaning the control loop can update 15 times per second.

According to NVIDIA, Cosmos 3 Edge ranks first on VANTAGE-Bench for vision analytics among models of a similar 4-billion-parameter size. The company also describes its robot-policy learning performance as state of the art. These claims position the model as a shared foundation for both smart-infrastructure applications, where systems analyze activity in the physical world, and robotics, where perception must be connected to concrete movement.

A world model does more than identify objects in an image. It learns a representation of how an environment changes over time, including objects, movement, spatial relationships, and the consequences of actions. For example, a robot asked to pick up an object must recognize it, estimate its position, track the motion of its gripper, anticipate what will happen when contact occurs, and choose the action most likely to complete the task. A world model can predict the visual result of an action, infer which action produced an observed change, or propose an action that should create a desired outcome.

Cosmos 3 Edge unifies the current state of the environment, possible future states, and the actions that connect them within a single on-device representation. This lets a physical AI system reason about what is happening now, simulate what may happen next, and decide how its behavior could influence that future. The same model can therefore be used for scene understanding, forward prediction, action inference, and policy generation instead of treating them as entirely separate problems.

The underlying Cosmos 3 architecture combines two transformer towers. An autoregressive tower processes visual and text tokens for understanding and step-by-step reasoning. A diffusion tower handles visual, audio, and action tokens for prediction, generation, and neural simulation. The towers retain separate normalization layers and multilayer perceptrons but share multimodal attention layers. Those shared layers align information across language, video, audio, and action, giving the system a common basis for interpreting instructions and physical changes.

The architecture also applies different attention patterns to different forms of information. Language tokens use causal attention, so each token can use the tokens that precede it. Diffusion tokens can attend more broadly to the available context, supporting coherent prediction and generation. Depending on the task, the autoregressive side can produce reasoning tokens, while the diffusion side can generate denoised video or action tokens. This arrangement allows Cosmos 3 to reason about a scene before producing a prediction or control output.

Physical machines describe movement in different ways: a vehicle may use its own pose and trajectory, a moving camera uses camera motion, a robot arm relies on the pose of its end effector, and a gripper must additionally represent whether it is open, closed, or grasping. Cosmos 3 maps these different embodiments into a common action representation. Actions are encoded as compact geometric vectors, creating a direct relationship between control signals and visible changes in the environment.

This connection makes generated video useful as more than a visual forecast. It can depict how the physical world is expected to change in response to a particular action, allowing training examples to capture motion, control, and cause and effect. When used as a policy, the model predicts both an action and its anticipated visual consequence: the current state goes in, while the recommended behavior and a simulation of the likely next state come out. Action information can also move in the opposite direction, enabling the model to infer an action from its observed effects.

NVIDIA is also releasing Cosmos 3 Edge Policy (DROID), a manipulation policy post-trained on the DROID dataset for pick-and-place tasks, together with scripts for additional post-training. The example instruction given by NVIDIA asks a robot to pick up a banana and place it on a plate. The accompanying tooling is intended to let developers adapt the model to practical manipulation tasks using infrastructure such as a small cluster of H100 GPUs or an NVIDIA DGX Station.