Machine Learning from Scratch

The Big Question

Suppose a language model has $7,000,000,000$ parameters. If each parameter is stored in 16-bit floating point, the weights alone require:

7\times10^9 \times 2\text{ bytes}=14\text{ GB}

Training needs more than weights. It may also store gradients, Adam optimizer states, activations, and temporary buffers. Under a simplified full-fine-tuning accounting, weights may be 14 GB, gradients another 14 GB, and Adam's two moment estimates can add roughly 28 GB more before activations are counted.

Do we really need to change all seven billion parameters to teach one new style, domain, or task?

Core Intuition

A large pretrained model is a vast existing map of useful behaviours. Full fine-tuning redraws the whole map. Parameter-efficient fine-tuning adds a small overlay that redirects behaviour for the new task.

The base model stays frozen. A small number of new parameters learn the update:

W_{\text{adapted}}=W_0+\Delta W

$W_0$ is the frozen pretrained matrix. $\Delta W$ is the learned task-specific modification. The PEFT question is whether that update can be represented much more cheaply than a full matrix.

Interactive Demo

PEFT adapter lab

Fine-tuning method

Transformer block sketch

attention q

trainable update

attention v

trainable update

mlp up

frozen base

mlp down

frozen base

norm

frozen base

output

frozen base

LoRA rank8Target groups2Base bits4

Trainable parameters

4,194,304

0.060% of 7B

Base storage

14.0 GB

weights only estimate

One 4096x4096 matrix

16,777,216

full update values

LoRA update for one matrix

65,536

rank 8

Adapter equation

h = W0 x + (alpha / r) B A x
W0 frozen, A and B trainable

PEFT preserves the expensive pretrained model and learns a small specialised update around it.

LoRA From First Principles

A transformer layer contains weight matrices. Let one frozen matrix be:

W_0\in\mathbb R^{d_{\text{out}}\times d_{\text{in}}}

Full fine-tuning learns every element of a full update $\Delta W$ . LoRA constrains that update to be low rank:

\Delta W=BA

A\in\mathbb R^{r\times d_{\text{in}}}\qquad B\in\mathbb R^{d_{\text{out}}\times r}\qquad r\ll\min(d_{\text{in}},d_{\text{out}})

The adapted layer computes:

h=W_0x+\frac{\alpha}{r}BAx

Only $A$ and $B$ are trained. $r$ is the rank. $\alpha/r$ scales the update.

Parameter Count

For a $4096\times4096$ matrix, a full update has:

4096\times4096=16,777,216

With LoRA rank $r=8$ , trainable parameters are:

4096\times8+8\times4096=65,536

\frac{65,536}{16,777,216}\approx0.00390625=0.39\%

LoRA trains about 0.39% as many values for that matrix. The update is cheaper because it can move the model along only $r$ independent directions.

Why is $BA$ low rank? For any input $x$ , $BAx=B(Ax)$ . The vector $Ax$ has only $r$ coordinates, so the result is a combination of at most $r$ columns of $B$ .

Tiny Forward Pass

W_0=\begin{bmatrix}1&0\\0&1\end{bmatrix}\quad A=\begin{bmatrix}1&-1\end{bmatrix}\quad B=\begin{bmatrix}0.2\\0.5\end{bmatrix}\quad x=\begin{bmatrix}3\\1\end{bmatrix}

Ax=2

BAx=\begin{bmatrix}0.4\\1.0\end{bmatrix}

W_0x=\begin{bmatrix}3\\1\end{bmatrix}

W_0x+BAx=\begin{bmatrix}3.4\\2.0\end{bmatrix}

LoRA leaves the original output in place and adds a learned low-rank correction. Commonly, one LoRA matrix is initialised randomly and the other to zero, so $\Delta W=0$ at the start and the model initially behaves like the base model.

QLoRA, Adapters, And Prompts

LoRA reduces trainable parameters, but the frozen base model still occupies memory. QLoRA stores the frozen base in low precision, often 4-bit, while training LoRA adapters at higher precision. The base is quantised and frozen; the adapter is trained.

q=\operatorname{round}\left(\frac{x-z}{s}\right)\qquad \hat x=sq+z

This uniform quantiser is only introductory intuition. Real QLoRA systems use more specialised formats, metadata, and compute paths.

Bottleneck adapters insert small modules into the network:

h'=h+W_{\text{up}}\sigma(W_{\text{down}}h)

Prompt tuning learns virtual input embeddings $[p_1,\ldots,p_m,x_1,\ldots,x_n]$ . Prefix tuning injects learned vectors into attention states across layers.

Method Comparison

Method	What changes?	Advantage	Limitation
LoRA	Low-rank weight updates	Strong quality-efficiency balance	Rank and target layers matter
QLoRA	Quantised base plus LoRA	Much lower base memory	Quantisation adds complexity
Adapters	Inserted bottleneck modules	Modular and expressive	Can add inference operations
Prompt tuning	Trainable input vectors	Extremely few parameters	Can be weaker on some tasks
Prefix tuning	Trainable attention prefixes	Controls multiple layers	More implementation complexity

Implementation

import math
import torch
from torch import nn

class LoRALinear(nn.Module):
    def __init__(self, base: nn.Linear, rank=8, alpha=16, dropout=0.0):
        super().__init__()
        assert isinstance(base, nn.Linear)
        self.base = base
        self.rank = rank
        self.scale = alpha / rank
        self.dropout = nn.Dropout(dropout)

        for param in self.base.parameters():
            param.requires_grad = False

        self.A = nn.Parameter(torch.empty(rank, base.in_features))
        self.B = nn.Parameter(torch.zeros(base.out_features, rank))
        nn.init.kaiming_uniform_(self.A, a=math.sqrt(5))

    def forward(self, x):
        base_out = self.base(x)
        update = (self.dropout(x) @ self.A.T) @ self.B.T
        return base_out + self.scale * update

    def trainable_parameters(self):
        return sum(p.numel() for p in self.parameters() if p.requires_grad)

base = nn.Linear(4096, 4096, bias=False)
lora = LoRALinear(base, rank=8)

x = torch.randn(2, 4096)
assert torch.allclose(lora(x), base(x), atol=1e-5)  # B starts at zero
print(lora.trainable_parameters())

from peft import LoraConfig, get_peft_model

config = LoraConfig(
    r=8,
    lora_alpha=16,
    target_modules=["q_proj", "v_proj"],
    lora_dropout=0.05,
)

model = get_peft_model(base_model, config)
model.print_trainable_parameters()

# Train normally, then save only adapter weights.
model.save_pretrained("medical-research-assistant-lora")

Interview Discussion

What stays frozen in LoRA?

The original pretrained weights. Only the low-rank adapter matrices are trained.

Why does LoRA save memory?

It avoids gradients and optimizer states for most base parameters.

What is the key difference between LoRA and QLoRA?

QLoRA also stores the frozen base model in low precision to reduce memory.

Is PEFT always as good as full fine-tuning?

No. Tasks requiring large representational changes may need more capacity.

Active Recall

1. Why is full fine-tuning memory expensive?

2. Write the LoRA adapted-layer equation.

3. Compute LoRA parameters for a 4096 by 4096 matrix with rank 8.

4. Why is the rank of BA at most r?

5. What is the difference between quantising the base model and training the adapter?