LiveEdit: Real-Time Streaming Video Editing Framework Accepted at ECCV 2026

Text-instruction-driven video editing is shifting from offline content production to online interactive scenarios such as live streaming effects, video conferencing, and augmented reality. However, high-quality video diffusion models typically rely on bidirectional spatiotemporal attention over the entire video, requiring the model to wait for future frames, which fails to meet low-latency, continuous-input, and instant-output requirements.

The team identified two major obstacles. First, switching from bidirectional to causal attention causes distribution shift, as the model can only access current and past frames. This dilutes attention over longer history and leads to forgetting of the original video structure, causing flicker or drift. Second, standard models recompute all spatial tokens, including unedited regions, wasting computation and potentially repainting static content.

LiveEdit uses a three-stage progressive distillation pipeline built on Wan2.1-T2V-1.3B, trained on 20,000 high-quality data pairs filtered from the Ditto-1M dataset. Stage one, Foundation Tuning, retains the full bidirectional DiT structure to build a strong editing prior. Stage two, Teacher Forcing, introduces chunk-wise causal attention where each video block accesses only current and historical blocks, aligning the model's behavior with causal constraints. Stage three, Distribution Matching Distillation, compresses the sampling steps from 100 to 4 and removes classifier-free guidance.

To further reduce computation, LiveEdit introduces an AR-oriented Mask Cache. It predicts the spatial regions likely to be edited in the current video block by comparing the difference between original video latents and edited results from the previous block. Tokens in unedited regions reuse cached intermediate features from the Self-Attention layers, while edited tokens receive full computation. The threshold for masking is dynamically adjusted based on token redundancy, pruning around 70% of redundant spatial tokens in the reported configuration. The cache is applied only to Self-Attention layers, as FFN features carry high-frequency spatial details that would cause blurring if reused.

In efficiency tests, LiveEdit processed an 81-frame video with a total latency of 7.89 seconds, achieving 12.66 FPS. The team also built a test set of 120 samples and evaluated six dimensions including text alignment, background consistency, motion smoothness, dynamics, aesthetic quality, and video quality. Qualitative results show that LiveEdit accurately performs local color, material, and attribute replacements with reduced drift to unedited regions. In a user study with 20 volunteers, LiveEdit achieved top-3 preference rates of 100.0%, 87.5%, and 95.8% for instruction consistency, background preservation, and overall quality, respectively.

The method relies on an assumption that edited regions remain relatively stable across adjacent video blocks. When targets move quickly, editing regions change drastically, or large-scale global modifications occur, the mask estimation and caching strategies still have room for improvement. Maintaining stability on longer videos, higher resolutions, and complex interactive instructions remains future work.