Machine Learning from Scratch

The Big Question

A 7B-parameter model stored in FP16 needs about 14 GB for weights alone. If each weight used 8 bits, that becomes about 7 GB. With 4 bits, about 3.5 GB, before metadata and runtime overhead.

Can we make every number smaller while preserving most of the learned function?

Core Intuition

Quantisation is like photograph compression. A photo with millions of colours can be reduced to a smaller palette and remain recognisable. Neural network weights can often be represented with fewer numerical levels and still behave similarly.

Quantisation compresses numerical representation. It does not deliberately teach a new behaviour the way fine-tuning does.

Interactive Demo

Quantisation lab

Bit depth4 bitsDynamic range[-2, 2]

Number line

7B storage

3.5 GB

Levels

MSE

0.006

x	q	x_hat	error
-1.90	0	-2.00	0.10
-1.20	3	-1.20	0.00
-0.70	5	-0.67	-0.03
-0.20	7	-0.13	-0.07
0.10	8	0.13	-0.03
0.55	10	0.67	-0.12
1.10	12	1.20	-0.10
1.80	14	1.73	0.07

Uniform Quantisation

A simple quantiser maps a floating-point value to an integer:

q=\operatorname{round}(x/s)+z

and reconstructs an approximation:

\hat x=s(q-z)

$s$ is the scale, $z$ is the zero point, and rounding creates error:

\epsilon=x-\hat x

\operatorname{MSE}=\frac1n\sum_i(x_i-\hat x_i)^2

Too few bits increase rounding error. Too narrow a range causes clipping. Too wide a range wastes levels on rare outliers.

PTQ, QAT, And LLM Methods

Post-training quantisation quantises after training. It is simple and does not require retraining, but can lose accuracy if the model is sensitive.

Quantisation-aware training simulates quantisation during training using fake quantisation. Because rounding is not usefully differentiable, implementations often use a straight-through estimator: forward pass rounds, backward pass passes an approximate gradient through.

LLM-specific methods such as GPTQ, AWQ, and NF4 are designed around transformer weight distributions, outlier channels, calibration data, and hardware constraints. The details differ, but the goal is the same: reduce memory and bandwidth while preserving behaviour.

Implementation

import numpy as np

def quantize_uniform(x, bits=8):
    qmin, qmax = 0, 2**bits - 1
    x_min, x_max = x.min(), x.max()
    scale = (x_max - x_min) / (qmax - qmin)
    zero_point = round(qmin - x_min / scale)
    q = np.round(x / scale + zero_point)
    q = np.clip(q, qmin, qmax).astype(np.int32)
    x_hat = scale * (q - zero_point)
    return q, x_hat, scale, zero_point

weights = np.array([-1.2, -0.2, 0.1, 0.8, 1.7])
q, restored, scale, z = quantize_uniform(weights, bits=4)
print(q)
print(restored)
print(((weights - restored) ** 2).mean())

# Template: loading an LLM in 4-bit when dependencies/hardware support it.
from transformers import AutoModelForCausalLM, BitsAndBytesConfig

config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype="bfloat16",
)

model = AutoModelForCausalLM.from_pretrained(
    "some-small-open-model",
    quantization_config=config,
    device_map="auto",
)

Interview Discussion

What does quantisation compress?

The numerical representation of weights or activations.

Why can quantisation speed inference?

Smaller values reduce memory bandwidth and can use specialised hardware kernels.

What is the difference between PTQ and QAT?

PTQ quantises after training. QAT trains while simulating quantisation.

What are common failure cases?

Outliers, too few bits, activation spikes, unsupported hardware, and unrepresentative calibration data.

Active Recall

1. How much storage do 7B weights need at FP16, INT8, and INT4?

2. Define scale and zero point.

3. What is quantisation error?

4. Why do outliers make quantisation harder?