Quantization-Aware Training (QAT)
Simulate quantization noise during training so the model learns weights that survive lower precision — better accuracy than PTQ at the same bit-width, especially below INT8.
Intent & Description
🎯 Intent
Train the model to tolerate the precision reduction it will face at inference time rather than applying quantization as a post-hoc surprise.
📋 Context
PTQ quantizes weights that were trained at full precision — the model never saw quantization noise during gradient updates. For aggressive targets (INT4, INT2) this mismatch degrades accuracy significantly. QAT bakes quantization into training.
💡 Solution
During forward passes, insert fake quantization nodes — rounding operations that simulate the INT4/INT8 grid — on weights and activations. Gradients still flow through fake-quant nodes using the straight-through estimator (treat rounding as identity for backprop). The model learns parameters that already cluster near quantization grid points. At deployment, real quantization is applied to a model that already expects it.
Real-world Use Case
📌 TL;DR
Train knowing you’ll quantize — fake quantization during training produces parameters that survive the precision drop far better than weights PTQ was never trained to handle.
Advantages
- Significantly better accuracy than PTQ at the same bit-width — especially at INT4 and below
- Robust to distribution shift — the model was trained expecting quantization noise
- Final weights are optimized for the actual inference precision, not retrofitted to it
Disadvantages
- Requires access to training pipeline and data — not a post-hoc transformation
- Training is slower due to fake quantization operations in every forward pass
- Hyperparameter sensitivity increases with more aggressive quantization targets