---
title: "LLM quantization explained: fewer bits, less memory"
description: "See how quantization shrinks a model's weight memory by storing FP16 values on a coarser grid of levels, and where the error actually lands."
published: 2026-07-18
updated: 2026-07-18
canonical: https://protrailblazer.com/posts/llm-quantization-explained/
tags: [machine-learning, demo, quantization, int8, int4, model-compression, llm]
---

# LLM quantization explained: fewer bits, less memory

**Answer:** Quantization shrinks an LLM by storing each weight with fewer bits, rounding the original FP16 values onto a small grid of representable levels. INT8 keeps 255 levels and halves the weight memory; INT4 keeps 15 levels and quarters it. The demo below quantizes a real 64x64 weight tensor live, so you can watch the memory collapse instantly, the rounding error creep in gradually, and a simulated next-token pick flip at the 2-bit extreme.

The interactive below runs real symmetric quantization over a seeded 64x64 weight tensor. Pick a precision, choose per-tensor or per-group scales, drag the morph scrub, and tap any cell in the matrix to read the exact original and quantized values side by side.

*Interactive demo: [open the quantization demo](https://protrailblazer.com/demos/quantization/)*

## What you're seeing

The scene runs left to right through four stages. On the left, a seeded FP16 weight field draws the original distribution as a histogram with sigma bands and a dashed mean line. Next, a quantizer gate shows the bin boundaries for the chosen precision, and particle tracers stream through it as the morph plays. To the right of that, a collapsing bit tower fills from the bottom as bits per weight drops, and a memory readout shows GB for 1B, 7B, and 70B weight sizes (labeled illustrative). The far-right panel is the simulated downstream head, a fixed 5-token prediction panel clearly marked SIMULATED / NOT A REAL LLM.

Below the main scene is a 64x64 tensor tile. The default view zooms in on a 12x12 patch where each cell is color-coded by rounding error from white at zero to deep red at maximum deviation. Switch to the full 64x64 view to see the global error pattern, then tap any cell to read the FP16 original, the quantized value, and the absolute difference.

The HUD across the top tracks bits per weight, unique representable levels, weight memory, compression ratio vs FP16, mean absolute error, and cosine similarity between the original and quantized tensor in real time.

A few things to try:

- **Switch the weight distribution to OUTLIERS with per-tensor INT4.** Watch mean absolute error jump versus the BALANCED preset, because a handful of large weights stretch the grid spacing and everything near zero rounds coarsely.
- **Keep OUTLIERS and switch from per-tensor to per-group.** The error drops visibly as each group gets its own scale. Check the HUD: effective bits per weight rises slightly (for example 4.13 at group size 128) because the scale values cost metadata.
- **Push precision to 2-bit and watch the downstream panel.** Symmetric 2-bit has only 3 representable levels. The top-1 token pick in the simulated head flips at this extreme, even though cosine similarity can still read near 1.0.
- **Drag the morph scrub slowly at INT4.** The bit tower collapses the moment you change precision, but MAE and cosine similarity move gradually with the morph, so the visual cost of each level of rounding emerges step by step.
- **Tap individual cells in the matrix.** The heaviest error cells are not random: they cluster where the original value sat between two widely spaced quantization levels.

## How quantization actually works

Every weight in an LLM is stored as a floating-point number, usually FP16 (16 bits, ~65,000 possible values per weight). Quantization replaces that with a lower-bit integer by computing a scale factor, dividing every weight by that scale, rounding to the nearest integer, and clamping to the integer range.

For symmetric quantization (what the demo uses), the scale is the largest absolute value in the tensor divided by the maximum representable integer. Every weight is divided by that scale, rounded to the nearest integer, and stored. At inference the stored integer is multiplied back by the scale to recover an approximation of the original. The error is whatever the nearest integer could not represent exactly.

INT8 gives 255 unique levels (from -127 to 127 in symmetric form) and halves weight memory relative to FP16. INT4 gives 15 levels and quarters memory. The 2-bit concept in the demo has only 3 levels (-1, 0, 1 in symmetric form), which is enough to illustrate breakdown but not a production format.

During a model's forward pass, the quantized weights are dequantized on the fly before each matrix multiply: the integer is multiplied by the stored scale and used as if it were the original FP16 value. Memory savings come from storage and transfer; compute precision depends on the kernel and hardware.

## Why per-group scales matter

With a single per-tensor scale, the largest weight in the entire tensor sets the grid spacing. If a few weights are five to nine times larger than the rest, the levels spread apart and everything clustered near zero rounds coarsely, wasting most of the 15 available INT4 levels on a range that has almost no weights in it.

Per-group quantization divides the weights into small blocks (commonly 128 weights) and gives each block its own FP16 scale. Each block's scale is sized for that neighborhood, so the grid fits the local range and rounding error drops. This is what makes INT4 usable in practice: methods like GPTQ, AWQ, and NF4 all exploit per-group (or per-channel) scales.

The cost is metadata. At group size 128, each group needs one FP16 scale stored alongside its 128 INT4 weights. That adds roughly 0.13 bits per weight, pushing effective storage from 4.00 to about 4.13 bits per weight. You can see this in the HUD when you enable per-group mode. The memory savings are still large, just not quite as clean as the raw bit count suggests.

Quantization also affects what the KV cache costs at inference time. Some deployments quantize the KV cache separately from weights, because the two have different sensitivity profiles. Reducing KV cache precision from FP16 to INT8 can nearly halve the memory required for long-context generation, though the tradeoffs differ by layer and model.

To see how the quantized weights feed into prediction, the [next-token generation walkthrough](https://protrailblazer.com/posts/how-llms-generate-text) traces the full path from weights through logits to a sampled token. And for the tradeoff between precision and generation speed at inference, see [inference-time compute](https://protrailblazer.com/posts/inference-time-compute-explained).

## Frequently asked questions

### What does quantization actually do to a model's weights?

It replaces each FP16 weight (roughly 65,000 representable values) with a lower-bit integer (255 for INT8, 15 for INT4) by dividing by a scale factor and rounding to the nearest integer. At inference, the integer is multiplied back by the scale. The rounding error is the price; the memory saving is the reward.

### Does INT4 quantization make a model meaningfully worse?

Often very little for INT4 with good per-group scales and careful calibration, but "INT4" is not a single fixed quality level. The same label covers different quality depending on architecture, calibration data, method (GPTQ, AWQ, NF4, GGUF), the precision kept for activations and the KV cache, and what you are evaluating. Bit count alone does not predict usability.

### What is the difference between per-tensor and per-group quantization?

Per-tensor uses one scale for the entire weight matrix, sized to the largest absolute value. Per-group divides weights into small blocks (commonly 128 per block) and gives each block its own scale. Per-group dramatically reduces error when a few outlier weights would otherwise stretch the grid spacing and waste most of the available quantization levels.

### Why does the demo's cosine similarity stay near 1.0 even when the token pick flips?

Cosine similarity measures the angle between the original and quantized tensor as vectors. Even at 2-bit, most weights move a little and the overall direction barely shifts, so cosine can read above 0.95. But the downstream softmax picks the token with the highest logit, and logit gaps between competing tokens can be tiny. A small absolute error in the right weight is enough to flip the pick even when the tensor as a whole still points roughly the same direction.

### How does quantization interact with the context window?

The context window length determines how many key and value vectors the model caches during generation. At FP16 those vectors are large; quantizing the KV cache to INT8 or INT4 can roughly halve or quarter that memory cost, which is one reason large-context deployments often quantize weights and KV cache separately.

## Key takeaways

- The memory win is arithmetic, not statistical. Dropping FP16 to INT4 cuts a 7B model's weight memory from 14 GB to 3.5 GB before you know anything about accuracy, which is why quantization decisions start with the bit count and end with the error measurements.
- One outlier weight can poison thousands of others. With a single per-tensor scale, the largest absolute value sets the grid spacing, so a few weights 5 to 9 times bigger than the rest stretch the levels apart and everything huddled near zero rounds coarsely.
- Per-group scales buy back fidelity with metadata, not magic. Each block of 16 to 256 weights gets its own FP16 scale, so effective bits per weight rise slightly (4.13 instead of 4.00 at group size 128) in exchange for a grid that fits each neighborhood.
- Cosine similarity can sit near 1.0 while behavior still shifts. In the demo's simulated head, the top-1 pick flips at the 2-bit concept setting even though most weights barely moved, because downstream decisions ride on small logit gaps.
- Bit count alone does not predict usability. The same INT4 label covers wildly different quality depending on architecture, calibration, method (GPTQ, AWQ, NF4), kernels, and what precision the activations and KV cache keep.