Machine Learning from Scratch

The Big Question

Give a raw pretrained completion model this prompt: Write a concise explanation of gradient descent.It may continue the text as if it were an exam page, a forum post, or a document.

An instruction-tuned model is more likely to answer directly: gradient descent is an optimisation method that repeatedly adjusts parameters in the direction that reduces loss.

Both models predict tokens. Why does one behave like an assistant?

Core Intuition

Pretraining is reading a vast library. Instruction tuning is practising with a teacher who repeatedly provides a task and a good response. The model already knows many facts, styles, and transformations; instruction tuning teaches it how to organise those abilities into useful responses.

This distinction is essential: instruction tuning is primarily about the training data and loss objective. PEFT is about which parameters are updated. You can do instruction tuning with full fine-tuning, LoRA, QLoRA, or another optimisation strategy.

Interactive Demo

Instruction tuning lab

Raw completion model

Write a concise explanation of gradient descent.
This question appeared on the exam last year and is often followed by examples...

Instruction-tuned model

Gradient descent is an optimisation method that repeatedly adjusts parameters in the direction that reduces the loss.

Instruction-tuning strength0.65Summarisation mixture45%Response-only loss maskingChat template matches inference

Masked token sequence

[user]add2and3[assistant]5

Prompt tokens condition the answer. Assistant tokens usually provide the direct loss signal.

Held-out task scores

QA87%

Summarise91%

JSON70%

Classify79%

The Objective Is Still Next-Token Prediction

A pretrained causal language model learns:

P(x_1,x_2,\ldots,x_T)=\prod_{t=1}^{T}P(x_t\mid x_{<t})

\mathcal L_{\text{pretrain}}=-\sum_{t=1}^{T}\log P_\theta(x_t\mid x_{<t})

This teaches grammar, document structures, facts, styles, and continuation behaviour. It does not uniquely teach which text is the user's request, where the assistant answer begins, which instruction has authority, or when to stop.

Instruction tuning changes the distribution and structure of the sequences. Many pipelines compute loss only on assistant response tokens:

m_t=\begin{cases}1 & \text{token }t\text{ belongs to assistant response}\\0 & \text{otherwise}\end{cases}

\mathcal L=-\sum_{t=1}^{T}m_t\log P_\theta(x_t\mid x_{<t})

Prompt tokens still enter the context. They condition the answer. But their own token prediction losses may be masked out.

Anatomy Of An Instruction Example

{
  "instruction": "Summarise the paragraph in one sentence.",
  "input": "Gradient descent updates parameters...",
  "response": "Gradient descent iteratively changes parameters to reduce loss."
}

A chat-formatted version might become:

<system>
You are a concise study assistant.
<user>
Summarise the following paragraph in one sentence:
Gradient descent updates parameters...
<assistant>
Gradient descent iteratively changes parameters to reduce loss.

Exact special tokens and templates depend on the model. The transformer receives one token sequence, not an abstract conversation object.

Concrete Dataset Walkthrough

Task	User request	Assistant target	Quality issue to watch
Explain	Explain regularisation simply.	Regularisation discourages unnecessary complexity so models generalise better.	Too vague or too long
Summarise	Summarise this paragraph in one sentence.	The paragraph says gradient descent updates parameters to reduce loss.	May omit key point
Classify	Classify sentiment: I loved the lecture.	positive	Label must be correct
Transform	Return JSON for name Ada and role tutor.	{"name":"Ada","role":"tutor"}	Format must be valid
Refuse	Answer using the missing article.	I do not have the article, so I cannot answer from it.	Should not hallucinate

Gradients On Assistant Tokens

For assistant target token $y_t$ , logits are $z_t\in\mathbb R^V$ :

p_{t,k}=\frac{e^{z_{t,k}}}{\sum_j e^{z_{t,j}}}

\ell_t=-\log p_{t,y_t}

\frac{\partial \ell_t}{\partial z_{t,k}}=p_{t,k}-\mathbb 1[k=y_t]

The correct token logit is pushed upward. Incorrect token logits are pushed downward in proportion to their current probability. With masking:

\frac{\partial \mathcal L}{\partial z_{t,k}}=m_t\left(p_{t,k}-\mathbb 1[k=y_t]\right)

If $m_t=0$ , the token contributes no direct loss gradient.

Why It Works

The model already has representations for task language, facts, styles, transformations, and dialogue patterns. Instruction examples repeatedly associate instruction patterns with appropriate response patterns.

P_{\theta'}(y_{\text{helpful}}\mid x)>P_\theta(y_{\text{helpful}}\mid x)

This is distributional adaptation, not symbolic rule installation. The model becomes more likely to complete request-shaped contexts with answer-shaped text.

Instruction Tuning Versus Nearby Ideas

Prompting	Changes inference-time context; does not change weights.
PEFT	Controls which parameters update; can be used to perform instruction tuning.
Full fine-tuning	Updates all parameters; can also use instruction data.
Preference optimisation	Learns from comparisons, not one target answer.
Alignment	Broader than obedience; includes safety, honesty, robustness, and values.

Data Quality And Mixtures

Instruction datasets may come from humans, benchmarks, transformed datasets, stronger models, self-instruction, or domain experts. Synthetic data can provide scale and variety, but it can also inherit errors, repeat styles, reduce diversity, or contaminate evaluation examples.

Task mixture matters. If 80% of examples are summarisation, the model practises summarisation far more than classification or structured output. If the data is narrow or optimisation is aggressive, instruction tuning can cause forgetting of general capabilities.

Implementation

def format_example(example):
    required = ["instruction", "response"]
    for key in required:
        if key not in example:
            raise ValueError(f"missing {key}")

    user = example["instruction"]
    if example.get("input"):
        user += "\n\n" + example["input"]

    return [
        {"role": "system", "content": example.get("system", "You are a concise study assistant.")},
        {"role": "user", "content": user},
        {"role": "assistant", "content": example["response"]},
    ]

def make_response_only_labels(input_ids, assistant_start, pad_token_id):
    labels = input_ids.copy()
    for i in range(len(labels)):
        if i < assistant_start or input_ids[i] == pad_token_id:
            labels[i] = -100  # ignored by PyTorch cross entropy
    return labels

tokens = ["[user]", "add", "2", "and", "3", "[assistant]", "5"]
labels = [-100, -100, -100, -100, -100, -100, "5"]

# Sketch: local-scale instruction tuning with LoRA
from peft import LoraConfig, get_peft_model

config = LoraConfig(r=8, lora_alpha=16, target_modules=["q_proj", "v_proj"])
model = get_peft_model(base_model, config)

# 1. format examples with the tokenizer chat template
# 2. tokenize
# 3. create response-only labels with -100 for prompt and padding tokens
# 4. train briefly
# 5. save adapter
model.save_pretrained("study-assistant-instruction-lora")

Interview Discussion

What changes in instruction tuning?

The model weights change, but the objective is still next-token prediction on instruction-response sequences.

Is instruction tuning the same as PEFT?

No. Instruction tuning is about the data and target behavior. PEFT is about updating only a small set of parameters.

Why mask prompt tokens?

The prompt should condition the answer, but the supervised target is usually the assistant response.

Does instruction tuning guarantee truthfulness?

No. It improves obedience and format adherence, but factuality, safety, and preference quality require additional evaluation and training.

Active Recall

1. Why can a raw pretrained model continue a prompt instead of obeying it?

2. What fields appear in an instruction-response example?

3. Write the response-only masked loss.

4. Explain instruction tuning versus PEFT.

5. Why can narrow instruction data cause forgetting?