Shengshu Technology Launches Vidu S2 With Real-Time Video Editing and Spatial Video

Vidu S2 includes two sub-models. Vidu S2-Avatar handles real-time interaction and supports voice dialogue, voice-controlled actions by a digital human, and reference images supplied during a session. Vidu S2-Editing handles real-time video editing and can change clothing, the person on screen, the background and the visual style while a video is playing.

The avatar model upgrades real-time output from 540p in the previous generation to 720p, making facial and clothing details richer during continuous interaction. QbitAI said its test compared Vidu S1 and S2 on the same dance, turn and stomp commands; S1 did not complete the requests, while S2 followed them. The new model can also receive a reference image mid-conversation, allowing the digital human to pick up an object shown in the image, wear a specified outfit or enter a new scene.

The editing model is the largest addition. In a QbitAI demonstration, a woman being interviewed changed into several outfits in real time, with the clothing appearing to fit her body and deform naturally with her movements. The report said a camel-colored coat partially covered jewelry at the cuff in a realistic way, but also noted limitations: a second outfit showed slight smearing in color blocks, and hair clipped through a hat when one was worn. The model can also swap backgrounds, replace the main subject and transfer visual styles.

These editing capabilities point to uses in livestreaming, virtual performances and creative video. QbitAI described a scenario in which viewers could ask a host to wear a dinosaur costume and see the change during the stream, turning viewers into directors and the host into a character that responds to live requests.

Vidu S2 also explores real-time spatial video. It can convert generated or edited video into left-eye and right-eye images for VR headsets, adding depth. Based on Vidu S2-Avatar, users can generate spatial video in real time; based on Vidu S2-Editing, they can edit spatial video in real time. An ordinary video can be turned into a spatial video, and the view from a headset camera of the physical world can be used as a canvas. The team acknowledged that headsets have strict resolution and latency requirements, and said work is continuing. It plans to expand from a fixed viewpoint to panoramic spatial video.

Vidu S2 arrived 69 days after Vidu S1. Released about two months earlier, Vidu S1 moved AI video from offline generation to real-time interaction, allowing a single uploaded photo to create an AI character that could interact with a user. QbitAI reported that users made cats talk, turned coding interfaces into a “programmer encourager” and revived deceased relatives. Vidu S1 reduced reliance on audio-driven lip sync and preset action libraries, enabling voice-controlled behavior and unlimited continuous interaction. The full-chain research and development for Vidu S1 and S2 was led by Zhang Jintao, a doctoral student of Tsinghua University professor Zhu Jun and head of streaming video generation and inference at Shengshu Technology.

Real-time interactive video computes each frame from user input, so generation speed must outpace playback. The team used SageAttention and multi-GPU scheduling to reduce waiting between modules. For training data, it added dance, 2D animation and 3D animation material and stabilized backgrounds while preserving large movements. Annotations record the order of events, when actions begin, what changes they bring and the state of the person afterward, with each time segment given corresponding conditions.

Streaming training is difficult because errors can accumulate from one segment to the next. Vidu S2 uses Hybrid Forcing to learn segment-by-segment generation, then Self-Replay Forcing to practice recovering from its own deviations. The method generates a continuous video, adds noise to blocks of the result and performs a trainable causal replay, connecting earlier and later segments so feedback from later segments can affect earlier representations. For image quality and real-time performance, Vidu S2 uses a Backbone-Refiner two-stage architecture. The Backbone generates low-resolution structure, motion and semantic content, while the Refiner adds details. An asynchronous pipeline lets detail reconstruction and subsequent content generation run in parallel, helping maintain frame rate at 720p output.

To change clothing or background during playback, Vidu S2 uses timestamp alignment. When a user sends a clothing image midway, the model must know not only what to change but also when the change should begin. Vidu S2 redesigns positional encoding to align the injection position of the reference image with a timestamp, so new conditions take effect from the correct point and maintain motion and lighting continuity. A VLM Agent based on a vision-language model continuously parses generated frames, tracks states, plans subsequent generation and adjusts conditions at suitable moments. Vidu S2 also adds preference and reward optimization in both bidirectional and streaming stages: DPO in the bidirectional stage and Streaming NFT in the streaming stage, with the replay approach used for reward optimization. The company said this improves image quality, expressions, action naturalness, audio-visual synchronization, action controllability and instruction following.