Optimising Predictive Models

Linear regression gave us a prediction rule. Mean squared error gave us a way to judge whether the predictions are good. The next question is practical.

We now have a loss function. How do we actually find the best parameters?

A parameter update is a small edit to the model. If the current weight makes predictions too low, we nudge it upward. If the current bias makes every prediction too high, we nudge it downward. Gradient descent is the rule that tells us which direction to nudge and how large the nudge should be.

We will use this tiny dataset to make the mechanics concrete. Predict, compute loss, compute a gradient, update the parameters, then repeat.

A derivative measures how much one quantity changes when another quantity moves a little. In optimisation, we care about how the loss changes when a parameter moves a little.

With one parameter, the derivative is the slope of the loss curve. With many parameters, the gradient collects all those slopes into one vector.

The gradient points in the direction where loss increases fastest. To reduce loss, we move the opposite way. The learning rate controls the size of that move.

Here $\eta$ is the learning rate. If it is too small, training crawls. If it is too large, the model can jump over good values and become unstable.

This lesson does not introduce backpropagation. Backpropagation is the efficient way neural networks compute gradients, and it belongs in Deep Learning. Here we are learning what an update means.

For one feature, linear regression predicts with a weight and a bias.

Mean squared error averages the squared residuals.

To update the weight, we need to know how the loss changes as the weight changes.

To update the bias, we ask the same question for the bias.

Suppose we begin with $w=0$ and $b=0$ on the three-house dataset. The predictions are all zero, so the residuals are the observed prices. The first weight gradient is:

With learning rate $\eta=0.01$, the first weight update is:

The parameter changed because the gradient said that increasing the weight would reduce the loss. Repeating this process keeps improving the model until updates become very small or validation performance stops improving.

Batch Choices

Batch gradient descent computes gradients using the complete dataset, averages them, and updates once.

Stochastic gradient descent uses one example, then updates immediately.

Mini-batch gradient descent uses a small group of examples, averages gradients inside that group, then updates once per batch.

Mini-batches dominate modern deep learning because they give a useful gradient estimate while letting hardware process many examples in parallel inside each batch. GPUs are especially good at doing the same operation over many examples at once.

This implementation trains the same linear model with batch, stochastic, and mini-batch gradient descent. The only difference is which examples are used to estimate the gradient before each update.

import numpy as np

def mse(X, y, w, b):
    predictions = X * w + b
    return np.mean((y - predictions) ** 2)

def gradients(X, y, w, b):
    n = len(X)
    predictions = X * w + b
    error = y - predictions
    dw = (-2 / n) * np.sum(X * error)
    db = (-2 / n) * np.sum(error)
    return dw, db

def train(X, y, method="batch", lr=0.01, epochs=100, batch_size=4):
    w, b = 0.0, 0.0
    history = []
    rng = np.random.default_rng(0)

    for epoch in range(epochs):
        indices = rng.permutation(len(X))

        if method == "batch":
            batches = [indices]
        elif method == "stochastic":
            batches = [[i] for i in indices]
        elif method == "mini_batch":
            batches = [
                indices[start:start + batch_size]
                for start in range(0, len(indices), batch_size)
            ]
        else:
            raise ValueError("Unknown method")

        for batch in batches:
            dw, db = gradients(X[batch], y[batch], w, b)
            w -= lr * dw
            b -= lr * db

        history.append(mse(X, y, w, b))

    return w, b, history

X = np.array([1, 2, 3, 4, 5, 6, 7, 8], dtype=float)
y = np.array([2.5, 4.2, 5.8, 7.1, 9.0, 10.5, 12.8, 14.2])

for method in ["batch", "stochastic", "mini_batch"]:
    w, b, losses = train(X, y, method=method, lr=0.01, epochs=200)
    print(method, round(w, 3), round(b, 3), round(losses[-1], 4))