Machine Learning from Scratch

The Big Question

We have learned how to train increasingly powerful models. A natural thought is: surely a bigger, more flexible model should always perform better.

Why does making the model more powerful sometimes make future performance worse?

The answer is the central problem of generalisation. Machine learning does not care about memorising the training table. It cares about performing well on new houses, new users, new images, new patients, or new requests drawn from the same underlying process.

We will use one running example: predicting house price from square footage. A line is too simple. A degree-10 polynomial can snake through every training point. Somewhere between those extremes is the model that actually generalises.

Core Intuition

Imagine studying for an exam. If you study too little, you do not understand the material. You fail both practice questions and new questions. That is underfitting.

If you only memorize the exact practice questions, you may look excellent on the practice sheet but fail the real exam. That is overfitting. The problem is not that you learned too much; it is that you learned accidental details instead of the pattern.

Underfitting

Model is too simple. Training and test error are both high.

Good fit

Model captures the stable pattern without chasing noise.

Overfitting

Model is too sensitive. Training error is low, test error is high.

Interactive Demo

Bias-variance lab

Housing example

Fit house price from square footage with a polynomial model. Higher degree means a more flexible curve.

Polynomial degree4Measurement noise18%Training examples35

Training error60

Test error35

Bias48

Variance19

Diagnosis

The model has enough flexibility to capture the real curve without chasing too much noise.

expected error = bias^2 + variance + irreducible noise
irreducible noise = 18

The Bias-Variance Decomposition

Suppose the true target is generated by:

y=f(x)+\varepsilon

where $\varepsilon$ is random noise with mean zero and variance $\sigma^2$ . For one input $x$ , imagine training the same learning algorithm on many different possible training sets. It produces many possible predictions $\hat f(x)$ .

The expected squared prediction error is:

\mathbb{E}\left[(y-\hat f(x))^2\right]

Substitute $y=f(x)+\varepsilon$ :

\mathbb{E}\left[(f(x)+\varepsilon-\hat f(x))^2\right]

Add and subtract the average learned prediction $\mathbb{E}[\hat f(x)]$ :

\mathbb{E}\left[\left(f(x)-\mathbb{E}[\hat f(x)]+\mathbb{E}[\hat f(x)]-\hat f(x)+\varepsilon\right)^2\right]

The cross terms vanish because the noise has mean zero and the deviations around $\mathbb{E}[\hat f(x)]$ average to zero. What remains is:

\mathbb{E}\left[(y-\hat f(x))^2\right]=\underbrace{\left(f(x)-\mathbb{E}[\hat f(x)]\right)^2}_{\text{Bias}^2}+\underbrace{\mathbb{E}\left[\left(\hat f(x)-\mathbb{E}[\hat f(x)]\right)^2\right]}_{\text{Variance}}+\underbrace{\sigma^2}_{\text{Noise}}

Bias is systematic error from being wrong on average. Variance is sensitivity to the particular training sample. Noise is irreducible error: measurement error, randomness, missing features, and human unpredictability that no model can fully learn.

Why Test Error Is U-Shaped

As model complexity increases, training error usually falls. A more flexible model can imitate the training data more closely. But test error behaves differently.

Low complexity: high bias dominates, so train and test error are both high.
Moderate complexity: bias falls without too much variance, so test error improves.
High complexity: variance dominates, so test error rises even while training error falls.

This is why training error is not enough. A model that perfectly fits the training data may simply be flexible enough to memorize noise.

Real Model Examples

Model	Typical risk	Concrete example
Linear regression	Often high bias on curved data	Straight line misses nonlinear housing patterns
Deep decision tree	Often high variance if unconstrained	Can memorize quirks of a small dataset
Neural network	Can be low bias, high variance without enough data or regularisation	Powerful enough to fit signal and noise
Random forest	Reduces variance by averaging trees	One reason ensembles work well

Implementation

import numpy as np
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import train_test_split

rng = np.random.default_rng(0)
X = rng.uniform(0, 10, size=(80, 1))
y = 50 + 8 * X[:, 0] - 0.7 * X[:, 0] ** 2 + rng.normal(0, 5, size=80)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.35, random_state=1)

for degree in range(1, 16):
    poly = PolynomialFeatures(degree=degree, include_bias=False)
    X_train_poly = poly.fit_transform(X_train)
    X_test_poly = poly.transform(X_test)

    model = LinearRegression()
    model.fit(X_train_poly, y_train)

    train_mse = mean_squared_error(y_train, model.predict(X_train_poly))
    test_mse = mean_squared_error(y_test, model.predict(X_test_poly))
    print(degree, round(train_mse, 2), round(test_mse, 2))

Repeat this with different training set sizes. With more examples, high-degree models become less unstable because the training sample gives fewer opportunities to chase accidental patterns.

Interview Discussion

What is high bias?

The model is too simple and is systematically wrong even on average.

What is high variance?

The model changes too much when trained on a different sample and often overfits.

Why can training error keep falling while test error rises?

The model is fitting idiosyncrasies of the training data rather than stable structure.

What is irreducible error?

Error from noise or missing information that cannot be eliminated by a better model.

Active Recall

1. Why is low training error not enough?

2. What does underfitting look like in train and test error?

3. What does overfitting look like in train and test error?

4. Why is test error often U-shaped as complexity increases?

5. What are bias, variance, and irreducible noise?