Nunchux AI Releases VC-Attention, a Training-Free Low-Bit Kernel for Video Diffusion Transformers

The kernel targets a bottleneck the company's researchers describe as structural. Video diffusion transformers flatten a clip into a single sequence of spatiotemporal tokens and run full self-attention at every layer. A five-second 720p clip on Wan2.2-14B spans about 70,000 tokens, and on the RTX 5090 attention consumes more than 64 percent of generation time. On a single B200, the team says attention accounts for roughly two-thirds of every MiniMax-H3 denoising step.

Two obstacles stood in the way of low-bit attention. Low-bit tensor cores accelerate the QK and PV matrix products, but prior methods such as SageAttention2 smooth queries and keys; after that smoothing and rotation, the value term accounts for 82 percent of output error on Wan2.2. The softmax between the two products also still runs in FP32, and on B200 and H200 the exponential and its FP8 cast become the longest pipeline stage. A Hadamard rotation preserves token norms and therefore does not remove the value outliers, which shift across heads, layers and steps; rotating V changed value error by only 0.2 percent.

V-Smooth attacks the outliers directly. An online k-means clusters value tokens per batch and head, and keys and values are permuted together so non-causal attention output is unchanged. Each 128-token hardware block then subtracts its mean, and only the residual is quantized with per-channel E4M3 at 8 bits or NVFP4 at 4 bits. The mean is added back using the row sum the online softmax already keeps, so no second pass or extra buffer is needed. Averaged over 100 Wan2.2 heads, the block mean removes 8 percent of block energy in sequence order, 12 percent under DeltaQuant's static cube and 36 percent after sorting. Each mean costs 0.125 bit per value element. Grouping runs only on the first 25 percent of denoising steps and the permutation is reused across four adjacent steps, which the company says costs 3 to 4 percent of attention time averaged over the full schedule.

ExpCast-FP8 addresses the softmax stage. Because an E4M3 byte read as an integer is roughly 8 log2(v) plus 56, the method writes the byte directly from the log-domain score with one fused multiply-add, using a constant of -0.35 that is not fitted per model. The direct path produces the same byte as the FP32 exponent-then-cast path on 79.6 percent of each doubling and lands one code away elsewhere. The paper proves a per-row total variation bound under 3.64 percent plus any underflow tail, and the measured average across 204,800 Wan2.2 attention rows is 1.6 percent. ExpCast-FP8 applies only to the 8-bit kernel because NVFP4 has no single affine log-to-code map. Hand-written CuTe/CUDA fusion of the preprocessing chain cuts one V-Smooth call from 42.2 milliseconds to 4.8 milliseconds on B200.

Benchmarks span four open-weight video diffusion transformers, Wan2.2-T2V-A14B, LongCat-Video, HunyuanVideo-1.5 and MiniMax-H3, with fidelity scored against BF16 FlashAttention-4 outputs over 100 prompts. On B200 at 8 bits, VC-Attention delivers a 1.59 times attention speedup and 1.19 times end-to-end; on H200, 1.46 times and 1.13 times. On workstation cards running 4 bits, the RTX PRO 6000 shows 2.27 times and 1.36 times, and the RTX 5090 shows 3.58 times and 1.70 times. On B200 the kernel is 6.02 times faster than SageAttention2, which ships no Blackwell kernel, while the H200 gap is 1.16 times. On 4-bit workstation cards V-Smooth matches SageAttention3 on the RTX PRO 6000 and stays within 5 percent on the RTX 5090, where fidelity separates the two. In fidelity terms, 8-bit V-Smooth adds 2.3 dB PSNR over SageAttention2 on Wan2.2 and 2.8 dB on HunyuanVideo-1.5; adding ExpCast-FP8 gives back 0.7 to 2.1 dB but still beats SageAttention2 on all four models. At 4 bits, V-Smooth beats SageAttention3 by 2.9 dB on Wan2.2 and 3.6 dB on LongCat-Video. The training-free Attn-QAT variant falls 3.4 to 6.7 dB below SageAttention2.

On MiniMax-H3 at 1344 by 768 resolution, attention runs 1.60 times faster than BF16 FlashAttention-4 on B200 with 20.2 dB PSNR against 19.9 dB for SageAttention2. The paper reports 1.47 times on B300 versus 1.31 times for a naive FP8 kernel, while the blog's chart lists 1.51 times for the same chip. Nunchux Attention, the company's proprietary extension, reaches 1.91 times on B200 and 1.83 times on B300 for MiniMax-H3 attention.

Because the method changes only the per-interaction cost, the company says it can be combined with sparse attention approaches such as Sparse VideoGen and Radial Attention, as well as with distillation and multi-GPU execution. No public kernel release has been announced; Nunchux runs the proprietary extension in its own stack and says free MiniMax-H3 access is coming through its Modelverse waitlist.