The Big Question
A classifier outputs 0.83. Should we approve the loan, diagnose cancer, block the transaction, or send the case to human review?
Where do we draw the line between negative and positive?
A classifier does not make decisions by itself. It produces scores. Threshold selection converts those scores into actions.
Core Intuition
A smoke detector has a threshold. Lower it and you catch more fires, but toast creates false alarms. Raise it and there are fewer false alarms, but some real fires are missed.
Medical diagnosis has the same shape. A low threshold catches more cancer cases but sends more healthy patients for follow-up. A high threshold reduces follow-up burden but misses more patients with cancer.
Interactive Demo
Medical decision threshold
Lower thresholds catch more cancer cases but create more false alarms. Higher thresholds reduce false alarms but miss more positives. The right threshold depends on the cost of each mistake.
Decision Rule
Suppose the model outputs a score interpreted as . A threshold converts the score into a label:
Sweeping from 0 to 1 produces many possible operating points. ROC curves plot false positive rate against true positive rate.
Precision-recall curves plot precision against recall and are often more informative for imbalanced problems.
If false negatives are very costly, the optimal threshold usually moves down. If false positives are very costly, it moves up.
ROC, AUC, And PR Curves
ROC AUC can be interpreted as the probability that a random positive example receives a higher score than a random negative example. It measures ranking quality across all thresholds.
But in rare-positive problems, ROC curves can look strong even when precision is poor. For the cancer dataset, thousands of true negatives make the false positive rate look small, while the actual number of false alarms may still be large. Precision-recall curves focus directly on positive predictions.
Business Examples
| Domain | Threshold pressure |
|---|---|
| Medical diagnosis | Lower threshold when missing disease is worse than extra testing. |
| Fraud detection | Balance fraud loss against human review cost and customer friction. |
| Spam filtering | Raise threshold if false positives hide important emails. |
| Search ranking | Choose cutoffs based on top-k user experience. |
| Autonomous driving | Thresholds must reflect safety-critical costs and uncertainty. |
Implementation
import numpy as np
from sklearn.metrics import roc_curve, precision_recall_curve, f1_score
probs = model.predict_proba(X_test)[:, 1]
thresholds = np.linspace(0.01, 0.99, 99)
best_f1 = None
best_cost = None
for t in thresholds:
preds = (probs > t).astype(int)
fp = ((preds == 1) & (y_test == 0)).sum()
fn = ((preds == 0) & (y_test == 1)).sum()
f1 = f1_score(y_test, preds)
cost = 5 * fp + 100 * fn
if best_f1 is None or f1 > best_f1[1]:
best_f1 = (t, f1)
if best_cost is None or cost < best_cost[1]:
best_cost = (t, cost)
fpr, tpr, roc_thresholds = roc_curve(y_test, probs)
precision, recall, pr_thresholds = precision_recall_curve(y_test, probs)Interview Discussion
Why not always use threshold 0.5?
The best threshold depends on costs, class imbalance, calibration, and business objectives.
What does ROC AUC measure?
Ranking quality: how often positives rank above negatives across thresholds.
When are PR curves better than ROC curves?
When positives are rare and precision on predicted positives matters.
How does calibration affect thresholding?
If probabilities are not calibrated, a score like 0.83 should not be interpreted literally as 83% risk.
Active Recall
1. Write the threshold decision rule.
2. What happens when you lower the threshold?
3. What are the axes of an ROC curve?
4. Why can ROC be misleading for imbalanced datasets?
5. How do false positive and false negative costs change threshold choice?