Calibration

Calibration asks whether predicted probabilities deserve their numerical interpretation.

The Big Question

Suppose a weather model makes 100 predictions with a 70% chance of rain. If the model is calibrated, it should rain on roughly 70 of those days.

Now replace rain with consequential predictions: a patient has 20% disease risk, a transaction has 90% fraud probability, a borrower has 7% default probability, or an image has 85% tumour confidence.

When a classifier says 80%, is it correct about 80% of the time?

Core Intuition

Calibration is about honesty of confidence. A bookmaker who repeatedly gives 2-to-1 odds should be right at the corresponding long-run frequency. A calibrated model is not always correct; a 60% prediction should still be wrong about 40% of the time.

Accuracy asks whether predictions are correct. Discrimination asks whether positives rank above negatives. Calibration asks whether confidence matches observed frequency.

Discrimination

Can the model rank risky patients above safer patients? Measured by ROC-AUC or PR-AUC.

Calibration

Among patients predicted at 0.8 risk, are about 80% actually positive?

Two models can have the same ranking but different calibration. Scores 0.99, 0.98, 0.97, 0.03, 0.02 and 0.70, 0.65, 0.60, 0.40, 0.35 may rank patients identically while making very different probability claims.

Interactive Demo

Calibration lab

Predicted risk versus outcome

predicted probability

Reliability diagram

Controls

ECE
0.120
MCE
0.500
Brier
0.196
BinCountConfidenceAccuracy
0.0-0.2280.110.25
0.2-0.4130.240.31
0.4-0.600.500.00
0.6-0.8130.750.38
0.8-1.0260.880.88

Main lesson

Calibration changes whether the score deserves its numerical interpretation. The ranking can stay almost the same while the reliability curve moves closer to the diagonal.

Formal Calibration

A perfectly calibrated binary classifier satisfies:

P(Y=1∣p^(X)=p)=pP(Y=1\mid \hat p(X)=p)=p

Exact probability values are rarely repeated, so we evaluate groups or bins. For bin BmB_m, define mean confidence and observed accuracy:

conf⁡(Bm)=1∣Bm∣∑i∈Bmp^i\operatorname{conf}(B_m)=\frac{1}{|B_m|}\sum_{i\in B_m}\hat p_i
acc⁡(Bm)=1∣Bm∣∑i∈Bmyi\operatorname{acc}(B_m)=\frac{1}{|B_m|}\sum_{i\in B_m}y_i

A reliability diagram plots confidence against accuracy. Points below the diagonal are overconfident; points above it are underconfident.

Numerical Walkthrough

Predictions: [0.1,0.2,0.2,0.4,0.4,0.6,0.7,0.8,0.9,0.9]
Outcomes: [0,0,1,0,1,1,1,0,1,1]

BinPredictionsOutcomesConfidenceAccuracyReading
0.0-0.30.1, 0.2, 0.20, 0, 10.1670.333underconfident
0.3-0.60.4, 0.40, 10.4000.500underconfident
0.6-1.00.6, 0.7, 0.8, 0.9, 0.91, 1, 0, 1, 10.7800.800close

For the first bin, confidence is (0.1+0.2+0.2)/3=0.167(0.1+0.2+0.2)/3=0.167and accuracy is (0+0+1)/3=0.333(0+0+1)/3=0.333. The observed positive rate is higher than the confidence, so the model is underconfident there.

Calibration Metrics

ECE⁡=∑m=1M∣Bm∣n∣acc⁡(Bm)−conf⁡(Bm)∣\operatorname{ECE}=\sum_{m=1}^{M}\frac{|B_m|}{n}\left|\operatorname{acc}(B_m)-\operatorname{conf}(B_m)\right|

Expected calibration error is a weighted average calibration gap. It depends on bin boundaries, number of bins, sample size, and class distribution, so it is an estimate, not a perfect truth measure.

MCE⁡=max⁡m∣acc⁡(Bm)−conf⁡(Bm)∣\operatorname{MCE}=\max_m\left|\operatorname{acc}(B_m)-\operatorname{conf}(B_m)\right|

Maximum calibration error highlights the worst bin.

Brier⁡=1n∑i=1n(p^i−yi)2\operatorname{Brier}=\frac{1}{n}\sum_{i=1}^{n}(\hat p_i-y_i)^2

Brier score penalises confident mistakes: prediction 0.9 with outcome 1 costs 0.01, while prediction 0.9 with outcome 0 costs 0.81.

L=−1n∑i=1n[yilog⁡p^i+(1−yi)log⁡(1−p^i)]\mathcal L=-\frac1n\sum_{i=1}^{n}\left[y_i\log \hat p_i+(1-y_i)\log(1-\hat p_i)\right]

Log loss punishes extreme confident errors even more. If y=1y=1 and p^=0.001\hat p=0.001, the loss is −log⁡(0.001)≈6.91-\log(0.001)\approx6.91.

Why Brier Encourages Honest Probabilities

Suppose the true probability is qq, but the model reports pp.

E[(p−Y)2]=q(p−1)2+(1−q)p2\mathbb E[(p-Y)^2]=q(p-1)^2+(1-q)p^2
=q(p2−2p+1)+(1−q)p2=p2−2qp+q=q(p^2-2p+1)+(1-q)p^2=p^2-2qp+q
ddp=2p−2q\frac{d}{dp}=2p-2q

Setting the derivative to zero gives p=qp=q. The second derivative is positive, so the expected Brier score is minimised by reporting the true probability.

Calibration Methods

Platt scaling fits a logistic regression from raw score s(x)s(x) to probability:

P(Y=1∣s)=σ(As+B)P(Y=1\mid s)=\sigma(As+B)

Isotonic regression learns a flexible non-decreasing mapping p=g(s)p=g(s). It preserves ordering but needs more data and can overfit small calibration sets.

Temperature scaling is common for neural networks:

P(Y=k∣x)=exp⁡(zk/T)∑jexp⁡(zj/T)P(Y=k\mid x)=\frac{\exp(z_k/T)}{\sum_j\exp(z_j/T)}

T>1T>1 softens probabilities. T<1T<1 sharpens them. Dividing logits by the same positive TT preserves ordering, so the predicted class usually stays the same while confidence changes.

Implementation

import numpy as np

def reliability_bins(probs, y, n_bins=10):
    bins = np.linspace(0, 1, n_bins + 1)
    rows = []
    for lo, hi in zip(bins[:-1], bins[1:]):
        mask = (probs >= lo) & (probs < hi if hi < 1 else probs <= hi)
        if mask.sum() == 0:
            continue
        conf = probs[mask].mean()
        acc = y[mask].mean()
        gap = abs(acc - conf)
        rows.append((lo, hi, mask.sum(), conf, acc, gap))
    ece = sum(count / len(y) * gap for _, _, count, _, _, gap in rows)
    return rows, ece
from sklearn.calibration import CalibratedClassifierCV

sigmoid = CalibratedClassifierCV(base_model, method="sigmoid", cv=5)
isotonic = CalibratedClassifierCV(base_model, method="isotonic", cv=5)

sigmoid.fit(X_train, y_train)
isotonic.fit(X_train, y_train)

probs_before = base_model.predict_proba(X_test)[:, 1]
probs_after = sigmoid.predict_proba(X_test)[:, 1]

Interview Discussion

Can a model have high AUC and poor calibration?

Yes. AUC measures ranking; calibration measures probability honesty.

Why use held-out data for calibration?

Calibrating on training or test data leaks information and produces optimistic estimates.

What does below the reliability diagonal mean?

Observed accuracy is lower than confidence, so the model is overconfident.

Does low ECE imply high accuracy?

No. A model can be honest about uncertainty while still making many errors.

Active Recall

1. State the formal calibration condition.

2. Explain discrimination versus calibration.

3. Compute confidence and accuracy for a reliability bin.

4. Why does ECE depend on binning?

5. Why can calibration fail across subgroups even if global ECE is low?