Machine Learning from Scratch

The Big Question

A classifier reports 92% accuracy, 0.89 F1, 0.94 ROC-AUC, and low calibration error. It looks strong in aggregate. Then you inspect the failures and discover it misses almost every night-time image, nearly all examples from one hospital, sarcastic comments, or rare product categories.

What is the model getting wrong, and why?

Aggregate metrics tell you whether there may be a problem. Error analysis tells you what the problem is.

Core Intuition

An 80% exam score tells us performance. It does not tell us which concepts were misunderstood, whether mistakes were careless or systematic, or what the student should study next. Error analysis is the machine-learning version of reviewing every missed question.

We will use an animal image classifier for cats, dogs, and foxes. Overall accuracy is high, but failures cluster around low lighting, blur, occlusion, and foxes that look like dogs.

Interactive Demo

Error-analysis workbench

FilterSort

Overall / low-light accuracy

42% / 0%

Selected example

#1: fox predicted as dog

Confidence: 0.97

Loss: 3.51

Lighting: low

Tag: low light

Failure categories

low light3

blur1

spurious background1

occlusion1

ambiguous1

Suggested intervention

Low-light foxes dominate the confident errors. A targeted next step is collecting more low-light fox images, adding brightness augmentation, and tracking the low-light slice separately in the next evaluation run.

Metrics Are Compressions

An aggregate metric compresses many outcomes into one number:

M:\{(\hat y_i,y_i)\}_{i=1}^{n}\rightarrow\mathbb R

Compression discards structure. Model A can miss all night-time images while Model B makes random errors across conditions. Both may score 90%, but their engineering implications are completely different.

For binary tasks, inspect false positives and false negatives separately:

\text{FP}:\hat y=1,\ y=0 \qquad \text{FN}:\hat y=0,\ y=1

In fraud detection, a false positive blocks a legitimate customer; a false negative permits fraud. They are different product failures.

Sort By Confidence And Loss

Confident errors are often the most informative. Predicting dog at 0.51 when the true label is fox is less alarming than predicting dog at 0.999.

\operatorname{severity}_i=\mathbb 1[\hat y_i\ne y_i]\cdot\max_k P(Y=k\mid x_i)

For binary cross-entropy, per-example loss is:

\ell_i=-\left[y_i\log p_i+(1-y_i)\log(1-p_i)\right]

High-loss examples reveal confident wrong predictions, mislabeled data, ambiguous examples, or distribution shift.

Error Taxonomy

Data-quality errors

Incorrect labels, duplicates, corrupted input, missing features, leakage, inconsistent annotation.

Representation errors

Relevant feature absent, poor encoding, insufficient resolution, tokenisation issue.

Model-capacity errors

Underfitting, inadequate architecture, inability to model interactions.

Optimisation errors

Poor convergence, unstable training, bad hyperparameters.

Distribution errors

Rare subgroup, dataset shift, out-of-distribution input.

Ambiguity

Multiple reasonable labels or unclear ground truth.

Decision-policy errors

Wrong threshold, wrong cost assumptions, poor calibration.

Slicing And Support Size

A slice is a meaningful subset: night-time images, one hospital, one age bracket, one language, rare classes, or long documents.

M_S=M\left(\{(\hat y_i,y_i):i\in S\}\right)

Example:

\operatorname{Accuracy}_{overall}=0.92\qquad \operatorname{Accuracy}_{night}=0.61\qquad \operatorname{Accuracy}_{day}=0.96

Support size matters. A slice with 50% accuracy over two examples is not equivalent to one with 50% accuracy over 2,000 examples. For an accuracy estimate $\hat p$ :

SE(\hat p)=\sqrt{\frac{\hat p(1-\hat p)}{n}}

If $\hat p=0.8,n=20$ , the standard error is about $0.089$ . If $n=2000$ , it is about $0.009$ .

From Diagnosis To Intervention

Diagnosis	Potential intervention
Low-light failures	Add low-light data or augmentation.
Label inconsistency	Relabel or clarify annotation guidelines.
Minority-class failure	Collect or reweight minority examples.
Threshold problem	Adjust the decision threshold.
Miscalibration	Calibrate probabilities.
Missing feature	Engineer or collect the feature.
Model too simple	Increase capacity.
Overfitting	Regularise or gather more data.
Distribution shift	Retrain on recent data.
Ambiguous task	Redefine labels or allow abstention.

Implementation

import pandas as pd
import numpy as np
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score

df = pd.DataFrame({
    "y_true": y_test,
    "y_pred": y_pred,
    "p_pred": probs.max(axis=1),
    "lighting": metadata["lighting"],
    "source": metadata["source"],
})

df["correct"] = df.y_true == df.y_pred
df["loss"] = -np.log(np.clip(df.p_pred.where(df.correct, 1 - df.p_pred), 1e-6, 1))
df["error_type"] = np.where(df.correct, "correct", "needs_review")

confident_errors = df[~df.correct].sort_values("p_pred", ascending=False)
low_light = df[df.lighting == "low"]

def evaluate_slice(frame, mask):
    subset = frame[mask]
    if len(subset) == 0:
        return {"n": 0}
    acc = accuracy_score(subset.y_true, subset.y_pred)
    se = (acc * (1 - acc) / len(subset)) ** 0.5
    return {
        "n": len(subset),
        "accuracy": acc,
        "standard_error": se,
        "precision": precision_score(subset.y_true, subset.y_pred, average="macro", zero_division=0),
        "recall": recall_score(subset.y_true, subset.y_pred, average="macro", zero_division=0),
        "f1": f1_score(subset.y_true, subset.y_pred, average="macro", zero_division=0),
    }

print(evaluate_slice(df, df.lighting == "low"))

The Error-Analysis Loop

train model
-> evaluate
-> inspect errors
-> create taxonomy
-> quantify categories
-> prioritise
-> make targeted change
-> rerun evaluation

Do this on a development set. If you repeatedly tune decisions from the final test set, the test set quietly becomes part of training.

Interview Discussion

Why are aggregate metrics insufficient?

They hide which examples, slices, and failure modes produce the errors.

Why inspect confident errors first?

They reveal cases where the model is not merely uncertain but confidently wrong.

What is slicing?

Evaluating meaningful subsets separately, such as low-light images or one hospital.

What if a prediction disagrees with the label?

The model may be wrong, the label may be wrong, the input may be ambiguous, or the problem definition may be flawed.

Active Recall

1. Why can two models with the same accuracy require different fixes?

2. How do false positives and false negatives differ in fraud detection?

3. What does a high-confidence error suggest?

4. Why should every slice report sample count?

5. Map three error categories to concrete interventions.

End Of The Unit

Generalisation & Model Selection has moved from why models fail, to controlling complexity, estimating performance, tuning settings, choosing metrics, selecting thresholds, checking probability honesty, and inspecting concrete failures.

A model is not good because one number is high. A trustworthy model generalises, uses an appropriate decision rule, expresses uncertainty honestly, and fails in ways we understand.