NVIDIA releases Cosmos3-H-Dreams for real-time generative surgical simulation

The release extends an earlier Cosmos-H-Dreams model, built on Cosmos-Predict2.5, which brought interactive generative simulation to dVRK tabletop suturing. NVIDIA said that earlier system learned visual dynamics directly from synchronized video and robot kinematics instead of explicitly modeling scene geometry, tissue deformation and physical interactions. Cosmos3-H-Dreams moves that approach from a controlled tabletop setting to video and kinematics recorded during clinical procedures on the CMR Surgical Versius platform.

NVIDIA described the clinical setting as substantially harder. In a tabletop environment, instruments operate against a relatively simple background. Inside the body, small instrument movements can affect an entire scene: tissue stretches under traction, reflections shift, structures become occluded and smoke changes visibility. Given an initial image and supplied tool actions, the simulator must capture those responses while preserving anatomy and instruments as it generates each new block of video. Small errors can accumulate and gradually alter instrument geometry or anatomical structure, making temporal consistency essential.

The company said visual realism is only a starting point for validation, not proof of biomechanical accuracy. The generated scene must also follow the controls: translation, rotation and gripper commands should produce coherent anatomical consequences. A plausible-looking rollout that ignores those commands cannot serve as an interactive environment, according to the blog post. Maintaining fidelity and responsiveness in a live control loop also requires low-latency generation. NVIDIA said Cosmos3-H-Dreams addresses that setting with a model specialized for clinical cholecystectomy and distilled for real-time generation.

Cosmos3-H-Dreams builds on Cosmos3-H-Embodied-Nano, a 16-billion-parameter model pretrained on Open-H-Embodiment. NVIDIA specialized a bidirectional teacher on curated cholecystectomy video and kinematics. Sample selection targets two visible, engaged instruments, a left Cadiere grasper and a right monopolar hook, and limits camera commands to emphasize instrument-driven interaction. The model uses 28 action dimensions: pose and gripper information for both arms, plus energy controls, thumbstick axes and thumbstick buttons. Training uses 832 x 480 video with actions aligned at 10 Hz. NVIDIA said that sampling rate is distinct from runtime generation throughput.

The training pipeline combines a teacher-to-student approach designed for long autoregressive rollouts with accelerated streaming inference through FlashDreams. NVIDIA first adapted the bidirectional teacher on 24 future frames, then extended the horizon to 72 frames, or 2.4 and 7.2 seconds at 10 Hz. Distillation then turns the bidirectional teacher into a causal student that can generate video in short blocks as new actions arrive. Causal teacher forcing prepares the student for sequential generation using recorded context. Self-Forcing then trains it on its own generated context, guided by the teacher and a learned critic, to help address error accumulation during interactive rollouts.

The first released student uses four denoising steps per generated block. NVIDIA then trained that student further with a two-step schedule to produce the second release checkpoint. The company said the two-step model is separately trained, not the four-step model run with fewer sampling steps. Fewer denoising steps yield faster inference at the expense of lower visual quality, and both checkpoints are intended to let developers explore the balance between visual quality and inference cost. FlashDreams supplies the accelerated streaming runtime. For long rollouts, the Cosmos3-H-Dreams integration uses an adapted FadeMem module that consolidates the growing key/value cache. NVIDIA said this complements Self-Forcing: distillation addresses how the model learns from its generated history, while cache consolidation addresses how that history is stored during streaming generation.

NVIDIA said a clinical procedure may involve complications or leave room for a surgeon to perform "novel" movements. A simulator therefore must generalize beyond its training data and produce coherent frames for unseen movements, the company said.