Pretraining

We now understand transformers, tokenisation, language modelling, and scaling. But there is still a practical question.

Suppose we want to build a language model from scratch. What actually happens?

Pretraining is the process of teaching a randomly initialized transformer general language patterns by exposing it to enormous amounts of text and training it with next-token prediction.

The Goal

Pretraining creates a foundation model. Like a child first learning language, facts, common sense, and patterns before specializing in medicine or programming, a pretrained model learns broad text structure before later fine-tuning or alignment.

Pretraining is where the whole course finally runs end to end. Data becomes tokens. Tokens become embeddings. The transformer produces logits. Cross-entropy measures error. Backpropagation computes gradients. The optimizer updates weights.

One update teaches the model almost nothing. Trillions of tokens and countless updates gradually create a model that captures grammar, facts, code patterns, style, and many forms of reasoning.

The Entire Pipeline

Data Collection

The first requirement is text. Modern language models train on mixtures of books, websites, encyclopedias, academic papers, code repositories, technical documentation, public forums, and multilingual text.

No manual question-answer labels are required. Ordinary text provides its own supervision because every next token is a target.

Data Cleaning

Raw internet text is messy. Pretraining data is filtered to reduce duplicated pages, spam, advertisements, boilerplate, corrupted documents, extremely repetitive text, and other low-quality material.

Dataset quality can matter almost as much as dataset size. A huge pile of duplicated or low-quality text teaches the model the wrong patterns.

Tokenisation And Examples

A sentence such as Jessie ordered a burger. becomes token IDs:

From a stream of token IDs, training examples are created by shifting the sequence:

One document becomes many supervised next-token prediction problems.

One Optimisation Step

A single training step looks like this:

If the model predicts probability $0.62$ for the correct token, the loss is:

Backpropagation computes $\frac{\partial L}{\partial W}$ for every parameter. The optimizer updates weights:

The model becomes slightly better. Then the process repeats.

Tokens Instead Of Epochs

In many ML courses, training is described in epochs: one pass through the dataset. LLM training often reports progress using tokens processed, optimization steps, and compute used instead.

The datasets are enormous, mixed, filtered, and sampled in ways that make a simple epoch count less informative.

Distributed Training

Frontier models cannot be trained on one GPU. Training is distributed across many machines using several forms of parallelism:

Distributed training changes how the computation is executed. It does not change the underlying objective: predict the next token and reduce cross-entropy loss.

What The Model Learns

Nobody labels examples as grammar, algebra, Python, or style. These abilities emerge because they help the model predict text. Over many updates, the model learns useful internal representations of language and knowledge.

What Pretraining Does Not Do

A pretrained model is not automatically a helpful assistant. It has learned to continue text. If prompted with a question, a raw pretrained model may continue in the style of internet text rather than provide a careful answer.

Instruction following and preference alignment come after pretraining.

A miniature pretraining loop uses the same ingredients as a large one, just at toy scale.

import torch
from torch import nn

model = TinyLanguageModel(...)
optimizer = torch.optim.AdamW(model.parameters(), lr=3e-4)
criterion = nn.CrossEntropyLoss()

for step, batch in enumerate(dataloader):
    # batch shape: (batch_size, sequence_length)
    inputs = batch[:, :-1]
    targets = batch[:, 1:]

    logits = model(inputs)

    loss = criterion(
        logits.reshape(-1, vocab_size),
        targets.reshape(-1),
    )

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

    if step % 100 == 0:
        print(step, loss.item())

The real version adds distributed training, checkpointing, data mixture scheduling, learning-rate schedules, monitoring, and fault recovery. But the core loop is the same.