Quantization Calculator
What int8, int4 and int2 do to model weights: the scale, rounding and error.
Questions
What is quantization?
Storing a model's weights as small integers instead of 16-bit floats, with one scale factor per group of weights. int8 halves the memory of fp16 and int4 quarters it, which is how large models fit on smaller GPUs.
How is the scale chosen?
This uses symmetric absmax quantization: the largest weight in the group maps to the largest integer (127 for int8, 7 for int4), and zero stays exactly zero. Real formats such as GPTQ, AWQ and GGUF use cleverer schemes, but start from this idea.
Why do small weights become 0?
At int4 there are only 15 levels, so a weight smaller than half a step rounds to 0. One large weight in a group makes the steps bigger for everyone, which is why real formats use small groups.
How much quality is lost?
Usually little at int8, a bit more at int4, and a lot at int2. The error shown here is for these weights only; the effect on a model's answers depends on the model and the format.
Related tools
- LLM Memory CalculatorHow much GPU memory a model needs: weights plus KV cache, with the maths shown.
- IEEE 754 Float InspectorThe bits and exact value of any float or double, such as 0.1 + 0.2.
- Positional Encoding VisualizerThe sine and cosine pattern a transformer adds to each position, as numbers and a heatmap.
- Softmax & Cross-Entropy CalculatorTurn scores into probabilities, and see the loss and perplexity for the right answer.