Generalisation & Model Selection
How do we know whether a model is actually good?
Evaluate, compare, tune, and debug trained models before putting them into production.
1. Bias versus Variance
Diagnose underfitting, overfitting, and the tradeoff between simple and sensitive models.
2. Regularisation
Discourage unnecessarily complex models by adding weight penalties to the loss.
3. Cross Validation
Estimate generalisation and compare model choices using held out folds.
4. Hyperparameter Tuning
Choose settings such as learning rate, tree depth, C, gamma, and regularisation strength.
5. Classification Metrics
Evaluate classifiers with confusion matrices, precision, recall, and F1.
6. Threshold Selection
Turn model scores into product decisions using thresholds, ROC curves, and precision recall curves.
7. Calibration
Check whether predicted probabilities mean what they claim.
8. Error Analysis
Inspect failures directly instead of trusting one aggregate score.