Definition
Quantization is the technique of storing a model's weights (and sometimes activations) at lower numeric precision — FP16 down to INT8 or INT4 — shrinking the memory footprint roughly proportionally: INT4 stores each weight in 4 bits instead of 16, a ~4x reduction that is often the difference between needing four GPUs and one.
Because LLM decode is memory-bandwidth-bound — the bottleneck is streaming weights and KV cache from GPU memory, not arithmetic — moving fewer bytes also makes inference faster, not just smaller. The cost is precision: aggressive quantization can measurably degrade output quality, so production teams evaluate the quantized model against the original on their own workloads before shipping, rather than trusting benchmark deltas.
How it comes up in interviews
The canonical capacity question: "this model doesn't fit on your GPU — options?" Quantization, smaller model, or more/larger GPUs; strong answers quantify (INT4 ≈ 4x smaller than FP16) and note the decode speedup follows from memory-bandwidth-bound inference, plus the eval-before-ship caveat.