New Method Creates 4-Bit Model That Outperforms Its Full-Precision Checkpoint

Typical efficiency pipelines compress the architecture, quantize the compressed weights, then apply a healing step. The dominant healing approach, quantization-aware training (QAT), injects fake-quantization operators into the forward pass and continues fine-tuning on a task loss. According to the blog post, QAT is costly because it reruns an already expensive multi-stage post-training process, and it can become unstable if training continues too long. Quantization-aware distillation (QAD), an alternative, distills a frozen full-precision teacher into the quantized student using KL divergence on output logits. The authors say QAD works when only quantization changes, because the same model exists in full precision. But after structural compression, there is no independently trained full-precision version of the smaller architecture; the only candidate teacher is the recovered bfloat16 checkpoint, itself a distilled approximation, which caps the quantized student's accuracy at that checkpoint's ceiling.

QAH removes that ceiling by distilling directly from the original, pre-compression model. Teacher and student need not share an architecture; the teacher remains full-size and full-precision while the student is half the size and runs in MXFP4. Because the teacher's output distribution is architecture-agnostic, the size and shape mismatch does not prevent transfer. The student never sees hard labels, only the teacher's output distribution, matched through KL divergence on logits.

This reframes quantization as a second, full pass of distillation against the original teacher, supervision that the bfloat16 checkpoint never received. The 4-bit student is not compensating for information lost to quantization, the blog post says; it is picking up information the earlier recovery stage did not transfer. There is also a stability benefit: since KL distillation ties the student to a fixed teacher distribution, once the student catches up there is no further pressure to drift, unlike a cross-entropy task loss that pushes toward hard labels indefinitely.

For long contexts up to 32,000 tokens, the authors reused a memory-efficient chunked KL-divergence loss from a companion paper on efficient distillation. That loss computes KL one slice of the sequence at a time and never materializes the full vocabulary-by-sequence grid, making the 32k-token healing fit inside a fixed GPU memory budget.

The paper reports that the QAH model wins on seven of the nine benchmarks it was tested on against the best full-precision version of the 60B architecture. The blog post emphasizes that this inverts the usual relationship between a 4-bit model and the 16-bit model it came from.