GGUF, GPTQ, AWQ and EXL2: Containers and Quantization Methods Explained

The explainer places safetensors, GGUF and PyTorch pickle files (.bin or .pt) in the container category, and GPTQ, AWQ, bitsandbytes NF4, and llama.cpp's K-quants and I-quants in the method category. EXL2 and EXL3 fall into both at once, because each is a quantization method plus a storage layout tied to one inference library.

Weight memory follows a simple arithmetic rule: parameters multiplied by bits per weight, divided by eight. On that basis an 8-billion-parameter model needs about 16 GB at 16-bit precision and about 4.5 GB at roughly 4.5 bits per weight, while a 70-billion-parameter model moves from about 140 GB to about 39 GB. The figures cover weights only; the KV cache and runtime overhead add more on top.

Unquantized models usually ship as 16-bit weights, either in pytorch_model.bin or model.safetensors. The older .bin and .pt files use Python pickle, and loading a pickle file can execute arbitrary code, which makes untrusted checkpoints a security risk. Safetensors, created at Hugging Face, removes that risk: a file is a small JSON header plus raw tensor buffers, with nothing executable inside, and tensors can be memory-mapped and loaded one at a time without reading the whole file. Safetensors is now listed as a PyTorch Foundation project. Most GPTQ, AWQ, EXL2, EXL3 and MLX models are also stored in .safetensors files, with the quantization living in the tensor contents and a config file rather than in a new container.

GGUF is a binary format for running models with GGML and GGML-based executors such as llama.cpp. Created by Georgi Gerganov, who also leads llama.cpp, it was introduced on August 21, 2023 as the replacement for the older GGML format. The earlier GGML, GGMF and GGJT files could not state which architecture a model belonged to, and adding a new hyperparameter broke every existing file; GGUF switched to typed key-value metadata so new fields can be added without breaking old files. Its specification lists five goals: single-file deployment, extensibility, mmap compatibility, easy loading, and complete information inside the file. Unlike tensor-only formats, GGUF can carry the tokenizer, special tokens and a Jinja chat template alongside the weights.

The suffix in a name such as Q4_K_M.gguf identifies the scheme. In the Hugging Face GGUF documentation cited by the explainer, Q2_K averages 2.625 bits per weight, Q3_K 3.4375, Q4_K 4.5, Q5_K 5.5 and Q6_K 6.5625, while the importance-matrix I-quant family runs from IQ1_S at about 1.56 bits through IQ2_XXS at 2.06 and IQ3_XXS at 3.06 to IQ4_XS at 4.25. The legacy 4-bit types Q4_0 and Q4_1 work out to 4.5 and 5.0 bits per weight, and Q8_0 to 8.5 bits.

The Q4_K figure comes from its block layout. A super-block holds 256 weights; 256 weights at 4 bits is 1,024 bits. Adding eight blocks of 12-bit scales and minimums contributes 96 bits, and a 16-bit super-scale plus a 16-bit super-minimum contributes 32 bits, giving 1,152 bits divided by 256, or 4.5 bits per weight. The _S, _M and _L suffixes denote mixes rather than new types: llama.cpp describes Q4_K_M as using Q6_K for half of the attention.wv and feed_forward.w2 tensors and Q4_K elsewhere, which is why a Q4_K_M file averages above 4.5 bits per weight. The documentation also lists TQ1_0 and TQ2_0 for ternary weights and MXFP4, a 4-bit microscaling floating-point type. Hugging Face files Q8_0 under legacy types, though the explainer notes it remains the standard near-lossless GGUF choice.

A reference table for a Llama-2-7B-class model shows the quality and size trade-off: FP16 scores 5.9565 perplexity at 13.0 GB, Q8_0 5.9584, a change of 0.03 percent, at 7.0 GB, Q6_K 5.9642, up 0.13 percent, at 5.5 GB, Q5_K_M 5.9796, up 0.39 percent, at 4.8 GB, and Q4_K_M 6.0565, up 1.68 percent, at 4.1 GB. The explainer states these numbers are illustrative, come from a 2023-era 7B model, and that newer models can react differently.

GGUF quantization can use calibration data. llama.cpp's llama-imatrix computes an importance matrix from a text file, and llama-quantize --imatrix uses it to improve quality; for 1-bit and 2-bit mixes, llama-quantize warns if no importance matrix is supplied. The specification defines filenames as base name, size label, fine-tune, version, encoding, type and shard, with shards using a five-digit counter such as 00003-of-00009 and optional mmproj- and mtp- prefixes marking vision projectors and multi-token-prediction draft modules. Hugging Face documentation lists llama.cpp, LM Studio, GPT4All and Ollama as environments where the format runs.