Machine Learning from Scratch

The Big Question

Bias and variance explained why overfitting happens. A flexible model can fit the training set by using fragile, overly specific patterns. Now we ask the natural next question:

Can we prevent a powerful model from becoming too complex?

We keep the same housing example. The model can use square footage, number of bedrooms, age, location features, and many derived interactions. Regularisation adds a cost for complexity so the model must justify every large weight.

Core Intuition

Imagine writing an essay with no page limit. You might ramble for 50 pages and include every minor detail. A page limit forces you to keep the ideas that matter.

Regularisation gives the model a complexity budget. It says: you may reduce training error, but large or unnecessary weights cost something. The optimizer is no longer minimizing prediction error alone.

\min_w L(w)+\lambda\Omega(w)

$L(w)$ is the ordinary prediction loss, $\Omega(w)$ is the complexity penalty, and $\lambda$ controls how strongly complexity is punished.

Interactive Demo

Regularisation lab

Penalty type

Regularisation strength lambda0.40

Learned housing features

size1.90

bedrooms-1.07

age0.54

garden0.21

zipcode hash-0.13

random id0.07

Training error

Test error

Complexity

Zero weights

What changed?

L2 shrinks all weights smoothly. The model keeps using most features, but less aggressively.

Increasing lambda usually raises bias and lowers variance. Too much regularisation turns overfitting into underfitting.

L2 And L1 Regularisation

L2 regularisation, also called ridge regression or weight decay in many neural network contexts, uses the squared length of the weight vector:

\Omega(w)=\lVert w\rVert_2^2=\sum_j w_j^2

J(w)=L(w)+\lambda\sum_j w_j^2

The gradient of the L2 penalty is:

\frac{\partial}{\partial w_j}\lambda w_j^2=2\lambda w_j

So a gradient step contains a term that pulls each weight back toward zero:

w_j \leftarrow w_j-\eta\left(\frac{\partial L}{\partial w_j}+2\lambda w_j\right)

L1 regularisation, also called lasso, uses absolute values:

\Omega(w)=\lVert w\rVert_1=\sum_j |w_j|

J(w)=L(w)+\lambda\sum_j |w_j|

Away from zero, the L1 penalty contributes approximately a constant push toward zero:

\frac{\partial}{\partial w_j}\lambda |w_j|=\lambda\,\operatorname{sign}(w_j)

That constant pressure is why weak weights can become exactly zero. L1 can perform feature selection; L2 usually shrinks weights but keeps most of them nonzero.

Why Regularisation Works

Regularisation connects directly to bias and variance. It often increases bias a little: the model is deliberately restricted. But it can reduce variance a lot: the model becomes less sensitive to accidental quirks in the training set.

no regularisation: low bias, high variance
moderate regularisation: slightly higher bias, much lower variance
too much regularisation: high bias, underfitting

This is not limited to linear models. Tree depth limits, minimum leaf sizes, dropout, early stopping, data augmentation, and label smoothing are all regularisation ideas: they restrict or stabilize the model so it generalises better.

Implementation

import numpy as np
from sklearn.linear_model import LinearRegression, Ridge, Lasso
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import train_test_split

rng = np.random.default_rng(0)
X = rng.normal(size=(120, 20))
true_w = np.array([4.0, -2.5, 1.2] + [0.0] * 17)
y = X @ true_w + rng.normal(0, 2, size=120)

X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=1)

models = {
    "linear": LinearRegression(),
    "ridge": Ridge(alpha=2.0),
    "lasso": Lasso(alpha=0.12),
}

for name, model in models.items():
    model.fit(X_train, y_train)
    train_mse = mean_squared_error(y_train, model.predict(X_train))
    test_mse = mean_squared_error(y_test, model.predict(X_test))
    zeros = np.sum(np.isclose(model.coef_, 0))
    print(name, round(train_mse, 2), round(test_mse, 2), "zero weights:", zeros)

Interview Discussion

What does lambda control?

It controls the strength of the complexity penalty relative to prediction loss.

Why does L2 shrink weights?

Its gradient is proportional to the weight, so large weights receive a stronger pull toward zero.

Why can L1 create sparse models?

Its absolute-value penalty can push small weights exactly to zero.

What happens with too much regularisation?

The model underfits because useful flexibility is also suppressed.

Active Recall

1. Write the general regularised objective.

2. How does L2 regularisation change a gradient step?

3. Why does L1 produce exact zeros more often than L2?

4. How does regularisation trade bias for variance?

5. Name three regularisation techniques outside linear models.