The Big Question
Suppose two cancer classifiers both achieve 99% accuracy. Which one is better? You do not know. In a dataset with 10,000 patients and only 100 cancer cases, a model can predict everyone is healthy and still be 99% accurate.
What does success actually mean for a classifier?
The answer depends on the cost of mistakes. Missing cancer is different from sending a healthy patient for an extra test. Metrics must reflect the objective, not mathematical convenience.
Core Intuition
Think of airport security. Missing a dangerous passenger is extremely bad. A false alarm is annoying and costly, but usually less catastrophic. Different errors have different consequences.
A confusion matrix names the four possible outcomes:
True positive
Predicted cancer and cancer is present.
False positive
Predicted cancer but patient is healthy.
False negative
Predicted healthy but cancer is present.
True negative
Predicted healthy and patient is healthy.
Interactive Demo
Cancer screening dataset
10,000 patients. 100 actually have cancer. Move the threshold and watch the confusion matrix and metrics change.
(TP+TN)/N
TP/(TP+FP)
TP/(TP+FN)
TN/(TN+FP)
harmonic mean
(recall+specificity)/2
Metric Definitions
Accuracy asks: what fraction of all predictions were correct? It can be misleading when classes are imbalanced.
Precision asks: when the model predicts positive, how often is it correct?
Recall asks: out of all actual positives, how many did we find?
Specificity asks: out of all actual negatives, how many did we correctly reject?
F1 is the harmonic mean of precision and recall. It punishes imbalance. If precision is 1.0 but recall is 0.1, the arithmetic mean is 0.55, but F1 is only .
Choosing The Right Metric
| Problem | Usually care about | Why |
|---|---|---|
| Medical screening | Recall | Missing disease can be catastrophic. |
| Spam filtering | Precision | Blocking legitimate email is painful. |
| Fraud detection | Precision-recall trade-off | Fraud is rare and review capacity is limited. |
| Search ranking | Precision@k | Users inspect only top results. |
| Class-imbalanced classification | Balanced accuracy or PR AUC | Accuracy hides minority-class failure. |
Implementation
from sklearn.metrics import (
confusion_matrix,
accuracy_score,
precision_score,
recall_score,
f1_score,
balanced_accuracy_score,
)
y_pred = model.predict(X_test)
print(confusion_matrix(y_test, y_pred))
print("accuracy", accuracy_score(y_test, y_pred))
print("precision", precision_score(y_test, y_pred))
print("recall", recall_score(y_test, y_pred))
print("f1", f1_score(y_test, y_pred))
print("balanced accuracy", balanced_accuracy_score(y_test, y_pred))Interview Discussion
Why can accuracy be misleading?
A majority-class predictor can look accurate while failing on the important minority class.
When is recall more important?
When missing positives is costly, such as medical screening.
When is precision more important?
When false positives are costly, such as spam filters or limited human review queues.
Why use F1?
It gives one number that rewards both precision and recall and penalises imbalance.
Active Recall
1. Define TP, FP, TN, and FN for medical diagnosis.
2. Why can 99% accuracy be useless?
3. What question does precision answer?
4. What question does recall answer?
5. Why does F1 use the harmonic mean?