Machine Learning from Scratch

The Big Question

Regularisation gave us a way to reduce overfitting. But now we have choices: no penalty, small penalty, large penalty, L1, L2, different polynomial degrees, different tree depths.

How do we know which model actually generalises best?

We continue the housing example. We will compare different regularisation strengths using held-out validation data. Cross validation is the engine that makes that comparison more stable.

Core Intuition

A single train/test split can be unlucky. Maybe the validation set contains unusually expensive houses. Maybe it misses small apartments. Maybe one neighborhood appears only in training. If you choose a model from that one split, you may choose the model that got lucky rather than the one that generalises.

Cross validation rotates the validation role. Every fold gets a turn as the held-out fold. The final score averages those validation scores.

fold 1: validate on D1, train on D2...Dk
fold 2: validate on D2, train on D1,D3...Dk
...
average validation scores

Interactive Demo

Cross-validation lab

Fold rotation

Number of folds5Split luck0

lambda	active fold	CV mean
0	78.3%	71.5%
0.1	80.6%	80.3%
0.5	84.3%	86.1%
1.5	76.3%	78.0%

Selected model

lambda = 0.5

Cross-validation chooses the setting with the strongest average validation score. It does not improve the model directly; it improves the reliability of the estimate used for model selection.

K-Fold Cross Validation

Split the dataset $D$ into $k$ folds:

D = D_1 \cup D_2 \cup \cdots \cup D_k

On round $i$ , train on all folds except $D_i$ and validate on $D_i$ . If the metric on that fold is $M_i$ , the cross-validation estimate is:

\operatorname{CV}=\frac{1}{k}\sum_{i=1}^{k}M_i

This average has lower dependence on one particular split. Cross validation does not make the model better directly. It makes our estimate of generalisation more reliable.

Choosing K

Choice	Strength	Trade-off
5-fold	Fast and common	Slightly noisier estimate
10-fold	More stable and still practical	More training runs
Leave-one-out	Uses almost all data for training each time	Very expensive and can have high estimate variance

Data Leakage

Cross validation only works if each validation fold is genuinely held out. Leakage happens when information from validation data sneaks into training.

Normalising the whole dataset before splitting.
Duplicate images appearing in both training and validation.
A patient appearing in multiple folds in a medical dataset.
Using future stock prices to build features for past predictions.
Choosing preprocessing rules after looking at validation outcomes.

The rule is simple: anything learned from data must be learned inside each training fold only, then applied to that fold's validation data.

Implementation

import numpy as np
from sklearn.linear_model import Ridge
from sklearn.model_selection import KFold, cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

alphas = [0.0, 0.1, 0.5, 1.0, 5.0]
kfold = KFold(n_splits=5, shuffle=True, random_state=0)

for alpha in alphas:
    model = make_pipeline(
        StandardScaler(),
        Ridge(alpha=alpha),
    )
    scores = cross_val_score(
        model,
        X,
        y,
        cv=kfold,
        scoring="neg_mean_squared_error",
    )
    mse_scores = -scores
    print(alpha, mse_scores.mean(), mse_scores.std())

# After choosing alpha, train on train+validation data.
# Evaluate once on the untouched test set.

Notice the pipeline. The scaler is fit separately inside each training fold, preventing validation statistics from leaking into training.

Interview Discussion

Does cross validation improve the model?

No. It improves the estimate used to compare models.

Why keep a final test set?

Because model selection uses validation scores. The test set should remain untouched until the final estimate.

What is nested cross validation?

An outer CV loop estimates final performance while an inner CV loop tunes hyperparameters.

How does CV connect to hyperparameter tuning?

Grid search, random search, and Bayesian optimisation use validation scores to choose settings.

Active Recall

1. Why can one train/test split give a misleading result?

2. Write the k-fold CV score formula.

3. Why should preprocessing happen inside each fold?

4. When should you use grouped or time-series cross validation?

5. Why does hyperparameter tuning need a validation procedure?

Common Mistakes

Time series: Use forward-chaining splits so the future never predicts the past.
Grouped data: Keep all examples from the same patient, user, or household in the same fold.
Class imbalance: Use stratified folds so class proportions remain similar.
Very small datasets: Variance remains high; report uncertainty and avoid overclaiming.