Scaling Laws

Language modelling gave us the objective: predict the next token and reduce cross-entropy loss. Now imagine training many language models with the same objective. One has 10 million parameters. Another has 10 billion. Another has hundreds of billions.

Can we predict how much better the larger model will be before we train it?

Scaling laws say that, over broad ranges, the answer is often yes. When model size, data, and compute increase, validation loss tends to fall in a surprisingly regular way. The improvement is not arbitrary. It often follows a power law.

This is one reason modern AI changed so quickly. Researchers learned that if the architecture was sound and the objective was simple, scale itself could be a reliable source of progress.

Suppose we train several transformer language models and measure validation loss. A toy set of results might look like this:

The loss keeps decreasing, but the gains get smaller. Going from 10 million to 100 million parameters helps a lot. Going from 10 billion to 100 billion still helps, but it costs far more for a smaller reduction in loss.

On ordinary linear axes, this looks like a curve that flattens. On log-log axes, the same relationship often looks close to a straight line. That straight-line behavior is the fingerprint of a power law.

A Simple Power Law

A simplified scaling law for model size might be written as:

Here $L(N)$ is validation loss, $N$ is the number of parameters, $A$ controls the scale, $\alpha$ controls how quickly loss improves, and $B$ is the irreducible floor approached by this trend.

The constants matter in research, but the course intuition is simpler: as $N$ grows, $N^{-\alpha}$ shrinks, so loss decreases. Because $\alpha$ is usually small, the decrease is gradual. Scale works, but it is expensive.

Three Things Scale Together

A language model is not improved by parameters alone. Three resources interact:

If a model is huge but sees too little data, it is undertrained. If the dataset is enormous but the model is too small, the model may not have enough capacity to absorb the patterns. If compute is too limited, training cannot fully use either one.

Compute-Optimal Scaling

Suppose two training runs cost roughly the same:

Compute-optimal scaling asks which allocation gives the best loss for a fixed compute budget. The Chinchilla result made this lesson vivid: many earlier large models were too big for the amount of data they saw. For the same budget, a smaller model trained on more tokens can outperform a larger undertrained model.

Why Bigger Models Can Do More

Bigger models have more representational capacity. They can store more patterns, form more specialized internal features, and combine those features in more ways. This does not make them intelligent by magic. It gives the optimization process a larger space in which useful circuits can form.

Emergent Abilities

Some abilities appear to become useful only after a model reaches a certain scale. A small model may fail at a task completely, while a larger model starts to solve it after seeing only a few examples in the prompt.

A sufficiently capable model can infer that the task is country-to-capital mapping and continue with Brasilia. This is in-context learning: the model changes its behavior from the prompt without changing its weights.

The word emergent should be used carefully. Some apparent jumps become smoother when researchers use better metrics or give partial credit. The safe claim is that scale can unlock qualitatively more useful behavior, even when the training objective stays next-token prediction.

Limits Of Scaling

Scaling is powerful, but it is not infinite. Larger models require more data, more compute, more memory, more engineering, and more energy. They can also be slower and more expensive to serve. Scaling laws help plan training runs, but they do not remove product constraints or guarantee trustworthy behavior.

A classroom version of scaling laws is simple: train several tiny language models, keep the dataset and training procedure comparable, then plot validation loss against parameter count.

import matplotlib.pyplot as plt

configs = [
    {"d_model": 64, "layers": 2},
    {"d_model": 128, "layers": 4},
    {"d_model": 256, "layers": 6},
]

results = []

for config in configs:
    model = TinyLanguageModel(
        vocab_size=tokenizer.vocab_size,
        d_model=config["d_model"],
        n_layers=config["layers"],
    )

    train(model, train_loader, steps=5000)
    val_loss = evaluate(model, validation_loader)
    params = sum(p.numel() for p in model.parameters())
    results.append((params, val_loss))

params = [row[0] for row in results]
losses = [row[1] for row in results]

plt.loglog(params, losses, "o-")
plt.xlabel("parameters")
plt.ylabel("validation loss")
plt.title("Tiny scaling experiment")
plt.show()

Real scaling-law research trains many more models, varies dataset size and compute, and fits power-law curves carefully. But the small experiment teaches the mental model: loss often improves smoothly enough that future training runs can be planned.