Nunchaku 4-bit Diffusion Inference Now Integrated into Diffusers

Nunchaku is an inference engine built around SVDQuant, a quantization technique that runs the main transformer layers with 4-bit weights and activations. Unlike weight-only quantization backends, which store weights in low precision but dequantize them at compute time, SVDQuant reduces memory usage and accelerates the denoising loop. With this release, quantized checkpoints can be loaded as easily as any other Diffusers model, and the companion diffuse-compressor toolkit allows users to quantize new architectures and publish them as regular Diffusers repositories.

To use the new integration, users install recent versions of diffusers, transformers, accelerate, kernels, and bitsandbytes. A pre-quantized pipeline can be loaded with ErnieImagePipeline.from_pretrained, followed by a standard inference call. The example in the announcement runs on an RTX 5090 and generates a 1024x1024 image in about 1.7 seconds with a peak memory usage of about 12 GB, compared with roughly 24 GB for the BF16 pipeline.

The NVFP4 checkpoints require NVIDIA Blackwell GPUs, while INT4 variants support earlier generations. Under the hood, Nunchaku Lite, the new integration path in Diffusers, patches the relevant nn.Linear modules of a stock Diffusers model with runtime SVDQ/AWQ linear layers before the checkpoint is loaded. The CUDA kernels are downloaded from the Hub through the kernels package. Two kernel families are used: svdq_w4a4 for attention and MLP projections, and awq_w4a16 for memory-bound and precision-sensitive normalization and modulation layers.

Because Nunchaku Lite does not rely on architecture-specific fused kernels and modules, it cannot match the full speedup of the original Nunchaku engine. However, the base implementation still delivers roughly 30% speedup while achieving the same level of VRAM reduction, making it a practical drop-in solution for Diffusers users seeking quantized diffusion models.