Hyperparameter Tuning

Hyperparameter tuning searches for the training setup that produces the best generalising model.

The Big Question

Cross validation gave us a way to estimate which model generalises best. But every algorithm still contains settings that are not learned automatically: learning rate, tree depth, number of trees, regularisation strength, SVM C, gamma, and k in k-NN.

How do we choose the settings that control training?

We will continue the housing dataset and train a random forest. The learned parameters are the tree splits. The hyperparameters are settings such as max_depth, n_estimators, and min_samples_leaf.

Core Intuition

Think of baking cookies. The recipe is the algorithm. Oven temperature, baking time, and sugar amount are hyperparameters. The finished cookie is what you get after the process runs.

Model parameters are learned during training: weights, tree splits, centroids, or neural network matrices. Hyperparameters are chosen before training and control how training behaves.

Parameters

Learned from data: weights, splits, centroids.

Hyperparameters

Chosen before training: learning rate, depth, C, gamma, lambda.

Interactive Demo

Hyperparameter tuning lab

Random forest housing model

Search strategy

Training score
79%
Validation score
92%
Complexity
27%

What the search is doing

Grid search evaluates a fixed lattice of settings. It is simple, but gets expensive quickly as dimensions multiply.

choose theta to maximize CV(theta)
theta = depth, trees, leaf size, learning rate, C, gamma, ...

Search As Optimisation

Let θ\theta represent a hyperparameter setting. For a random forest, it might include:

θ=(max_depth,n_estimators,min_samples_leaf)\theta=(\text{max\_depth},\text{n\_estimators},\text{min\_samples\_leaf})

Cross validation turns each setting into a score. Hyperparameter tuning searches for:

θ∗=arg⁡max⁡θCV(θ)\theta^*=\arg\max_{\theta} CV(\theta)

The challenge is that each evaluation is expensive: train the model, validate it, repeat across folds, then try another setting. A search over ten values for each of five hyperparameters already contains 10510^5 candidates.

Search Strategies

Grid search tries every point on a fixed grid. It is simple and reproducible, but it explodes combinatorially.

Random search samples settings from distributions. It often works better than grid search when only a few hyperparameters matter, because it explores more unique values along the important dimensions.

Bayesian optimisation uses previous trials to build a surrogate model of the search space. It then chooses new trials that balance exploration of uncertain regions with exploitation of promising regions.

Hyperband and early stopping allocate resources adaptively. Poor candidates are stopped early; promising candidates receive more training budget.

Nested Cross Validation

Hyperparameter tuning can overfit the validation procedure. If you try hundreds of settings, some setting may win partly by luck.

Nested cross validation separates selection from final estimation:

outer fold: estimate final generalisation
inner fold: tune hyperparameters
repeat for every outer fold

This is expensive, but it gives a more honest estimate when model selection itself is complex.

Implementation

from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import GridSearchCV, RandomizedSearchCV

model = RandomForestRegressor(random_state=0)

grid = {
    "max_depth": [4, 8, 12, None],
    "n_estimators": [50, 100, 200],
    "min_samples_leaf": [1, 3, 5, 10],
}

search = GridSearchCV(
    model,
    param_grid=grid,
    cv=5,
    scoring="neg_mean_squared_error",
)
search.fit(X_train, y_train)

print(search.best_params_)
print(-search.best_score_)
random_search = RandomizedSearchCV(
    model,
    param_distributions=grid,
    n_iter=20,
    cv=5,
    scoring="neg_mean_squared_error",
    random_state=0,
)
random_search.fit(X_train, y_train)

Interview Discussion

What is the difference between a parameter and a hyperparameter?

A parameter is learned during training. A hyperparameter is chosen before training and controls the training process or model family.

Why can random search beat grid search?

If only a few dimensions matter, random search explores more distinct values along those dimensions.

What does Bayesian optimisation add?

It uses previous trials to choose more informative future trials.

What is search overfitting?

Choosing a setting that wins validation by luck after trying many candidates.

Active Recall

1. Name three model parameters and three hyperparameters.

2. Write hyperparameter tuning as an argmax over cross-validation score.

3. Why does grid search become expensive quickly?

4. Why does early stopping save compute?

5. Why might nested cross validation be needed?