Fine-tuning

Pretraining teaches a transformer a broad skill: given the tokens so far, predict the next token. That objective can teach grammar, facts, style, code patterns, and many regularities in language.

How do we teach a pretrained language model to perform a specific job without training it from scratch?

Fine-tuning is the next stage. It starts with pretrained weights and continues training on a smaller, more specific dataset. The goal is not to relearn language. The goal is to bend an already capable model toward a target behavior.

Pretraining asks: what patterns exist in language? Fine-tuning asks: which of those patterns do we want this model to use?

Imagine Jessie has read recipes, reviews, menus, novels, textbooks, code, customer support chats, and random forum posts. She knows a lot about language and food. But if a user asks what to put on a cheeseburger, she might continue like a blog article, a forum thread, or a dataset entry.

The model did not lack all the knowledge. The missing piece was behavior. Fine-tuning shows the model many examples of the kind of continuation we want: direct answers, useful formatting, constraint-following, and task-specific style.

This works because the model does not start from random weights. Pretraining has already built rich internal representations. Fine-tuning adjusts those representations instead of creating them from nothing.

Pretraining Versus Fine-tuning

A supervised fine-tuning dataset usually contains examples of desired input-output behavior. For an assistant, each example may contain an instruction, optional input, and response.

{
  "instruction": "Suggest a burger using these ingredients.",
  "input": "beef, buns, onion, cheese",
  "response": "You can make a simple cheeseburger with fried onion and melted cheese."
}

That structured example is converted into one formatted training sequence:

Modern chat models often use special chat templates instead of visible markdown markers:

The exact format matters. The model learns patterns from the template. If training examples consistently mark where the assistant answer begins, the model learns to produce assistant-style text after that marker.

Fine-tuning usually uses the same next-token cross-entropy machinery as pretraining. Let the token sequence be:

At position $t$, the model predicts a distribution over the next token:

The token-level negative log likelihood is:

Averaged over a full sequence, the loss is:

Concrete Token Example

If the correct next token is You and the model assigns probability $0.70$, the loss is:

If it assigns probability $0.01$, the loss is:

Gradient descent updates the model so desired response tokens become more likely in the same contexts next time. There is no separate obedience module. The desired behavior becomes more probable under the model.

Masking The Loss

In assistant fine-tuning, the user message is context. The assistant answer is the target. We often compute loss only on assistant tokens.

If $m_t=0$, the token contributes nothing to the loss. In many PyTorch language-modeling setups, ignored labels are stored as -100.

Full Fine-tuning

Full fine-tuning updates all model parameters. If the pretrained model starts at $\theta_0$, training searches for:

Every attention matrix, feed-forward matrix, embedding matrix, and output matrix can change. This gives maximum flexibility, but it is expensive and creates a complete new copy of the model.

Parameter-efficient Fine-tuning

Parameter-efficient fine-tuning freezes most of the pretrained model and trains only a small number of new parameters. The base model already knows language. The adapter steers that model toward a task.

LoRA

LoRA, short for low-rank adaptation, freezes a large weight matrix $W$ and learns a small update $\Delta W$:

Instead of learning every entry in $\Delta W$, LoRA represents it as a low-rank product:

If $d=4096$ and $r=8$, the full matrix has:

The LoRA update has:

That is dramatically smaller. LoRA often works because fine-tuning does not need to invent a new language model. It mostly needs to steer an existing one.

Method Trade-offs

Fine-tuning Is Usually About Behavior

Fine-tuning can teach some domain information, but it is usually strongest for behavior, format, style, task structure, and domain habits. If a restaurant menu changes every week, retrieval is usually better because retrieval changes the context at inference time. Fine-tuning changes the model weights.

Prompt First

Prompting changes the input. Fine-tuning changes the model. Prompting is cheaper and easier to iterate on, so a practical rule is: prompt first, fine-tune when prompting repeatedly fails in a measurable pattern.

Data Quality

Fine-tuning amplifies the dataset. If examples are verbose, the model becomes verbose. If examples ignore constraints, the model learns to ignore constraints. A small clean dataset can beat a large noisy one.

Coverage

The dataset should include the important situations the model will face: direct questions, ambiguous requests, constraint-heavy requests, unsafe requests, format-specific requests, corrections, out-of-domain questions, and multi-turn conversations.

Overfitting And Forgetting

Too many epochs can make the model memorize surface quirks. If every answer begins with Great question!, the fine-tuned model may say that everywhere. Aggressive fine-tuning can also cause catastrophic forgetting, where the model gets better at the target task but worse at general abilities.

Fine-tuning usually uses a smaller learning rate than pretraining because the pretrained model is already valuable. The aim is to adapt enough, but not overwrite useful behavior.

Fine-tuning is best treated as an experiment loop, not a one-shot ritual.

The most important step is defining the target behavior. Make the model better is too vague. Answer burger recipe questions in fewer than 120 words, respect ingredient constraints, mention food safety when relevant, and use numbered steps for procedures is measurable.

The key object is the labels tensor. User tokens are often masked with -100, while assistant tokens remain as targets.

import torch

optimizer = torch.optim.AdamW(model.parameters(), lr=2e-5)

for batch in dataloader:
    input_ids = batch["input_ids"]
    attention_mask = batch["attention_mask"]
    labels = batch["labels"]  # user positions are -100

    outputs = model(
        input_ids=input_ids,
        attention_mask=attention_mask,
        labels=labels,
    )

    loss = outputs.loss
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()

Conceptually, the labels might look like this:

Validation loss matters, but it is not enough. A model can have lower loss and still produce bad assistant behavior. Evaluate actual outputs on normal requests, ambiguous requests, adversarial requests, domain edge cases, safety-sensitive prompts, multi-turn conversations, and format-specific prompts.

Useful grading dimensions include helpfulness, correctness, constraint-following, format adherence, calibration, safety, and brevity. Always compare against the base model and against a strong prompt-only baseline.