Machine Learning from Scratch

Six Months Later

At launch

Precision 94% · Recall 91%

Six months later

Abuse remains visible, moderator queues grow, and creators complain.

No code, weights, or thresholds changed. The world changed: new slang appeared, language mix shifted, and adversaries learned spellings the model misses.

Every deployed model eventually degrades. Monitoring should detect it before users have to.

Retrieval Warm-Up

Why can a healthy P50 coexist with a broken P99?
Which versions must be logged with every prediction?
What is the difference between shadow and canary traffic?

Reveal after retrieving

Tail requests can queue or cold-start; model, feature, policy, schema, and service versions; shadow has no user effect while canary decisions are real.

Learning objectives

Design telemetry and dashboards
Monitor systems, data, models, and product outcomes
Distinguish data, concept, prediction, and calibration drift
Quantify distribution movement
Set actionable alerts
Run a root-cause investigation

Production Incident Lab

Inject a production change

Diagnosis: No material change detected.

P99 latency

142 ms

Availability

99.96%

Error rate

0.12%

GPU util.

64%

Time series: model recall

M10

Observe The Whole Decision Path

Users → API → feature pipeline → model → decision
↓
request trace + feature statistics + prediction + outcome
↓
dashboard → alert → on-call engineer → investigation

Metrics summarise trends; logs record discrete events; traces connect one request across services. Every telemetry event needs event time, request ID, model/feature/policy versions, slice metadata, fallback state, and privacy controls.

Four Layers Of Health

Layer	Signals	Question
System	P50/P95/P99 latency, availability, errors, queue, CPU/GPU, memory	Can the service answer reliably?
Data	Schema, missingness, ranges, categories, feature distributions, freshness	Are inputs valid and familiar?
Model	Score distribution, confidence, delayed precision/recall, calibration, slices	Are predictions behaving and eventually correct?
Business	Reports, appeals, moderator workload, time-to-removal, creator satisfaction	Is the product outcome improving?

A green system dashboard can hide a harmful model. A stable prediction histogram can hide worsening user outcomes. Monitoring must join all four layers.

Drift Means The Distribution Moves With Time

Training approximated $P_{train}(x,y)$ . At time $t$ , production generates $P_t(x,y)$ . Monitoring estimates:

\Delta_t=P_t(x,y)-P_{train}(x,y)

Data drift changes $P(x)$ : average comment length grows from 18 to 34 words. Label shift changes $P(y)$ . Concept drift changes $P(y\mid x)$ : the same phrase acquires an abusive meaning. Unlabelled telemetry can detect input movement, but only delayed labels or outcome proxies reveal concept drift.

Measuring Drift

KL divergence measures how surprising production distribution $Q$ looks under reference $P$ :

D_{KL}(P\|Q)=\sum_i P_i\log\frac{P_i}{Q_i}

It is asymmetric and becomes unstable when $Q_i=0$ . Jensen–Shannon divergence compares both to their mixture $M=(P+Q)/2$ and is symmetric:

D_{JS}(P,Q)=\tfrac12D_{KL}(P\|M)+\tfrac12D_{KL}(Q\|M)

Population Stability Index is common for binned operational features:

PSI=\sum_i(Q_i-P_i)\ln\frac{Q_i}{P_i}

Worked PSI. Training comment lengths occupy short/medium/long bins [0.50, 0.35, 0.15]; production is [0.30, 0.40, 0.30]:

PSI=(-.20)\ln(.30/.50)+(.05)\ln(.40/.35)+(.15)\ln(.30/.15)\approx.213

That exceeds a common heuristic warning level of 0.2, but thresholds must be calibrated to each feature and business risk. Earth Mover’s Distance asks how far probability mass must move and respects ordered bins.

Prediction Collapse And Calibration Drift

If toxic predictions jump from 11% to 80%, investigate upstream schema/defaults, traffic mix, attack traffic, and model version. A stable rate is not proof of quality; a real event can legitimately move prevalence.

Calibration asks whether comments predicted at 90% toxicity are toxic about 90% of the time. With delayed reviewed outcomes, compute bins:

gap_k=\left|\operatorname{mean}(y_i\mid p_i\in B_k)-\operatorname{mean}(p_i\mid p_i\in B_k)\right|

Calibration can drift even if ranking remains useful, changing moderator workload and the meaning of fixed thresholds.

Root Cause Before Remedy

1. validate alert and user impact
2. scope: region, language, version, time, fallback
3. system: errors, queues, latency, resources
4. data: schema, missingness, freshness, drift
5. features: online/offline parity and versions
6. model: predictions, calibration, delayed labels, slices
7. product: policy, reports, appeals, moderator behaviour
8. mitigate → identify cause → prevent recurrence

Example: recall falls only for Spanish after 14:00 UTC. System health is green. Spanish traffic share doubled and new slang drives errors. This is not a GPU incident; route uncertain Spanish cases to review, collect labels, and prepare a candidate.

Alerts Must Be Actionable

Page now

P99 > 200 ms for 10 min and SLO burn is rapid.

Page now

Schema mismatch or GPU fleet unavailable.

Ticket/investigate

PSI > 0.2 for two windows with adequate volume.

Ticket/investigate

Prediction distribution moves beyond expected seasonality.

Do not page alone

One sparse slice produces a single noisy point.

Composite alert

Recall proxy falls while reports and moderator queue rise.

Tune thresholds, windows, severity, owner, runbook, and recovery condition. Alert fatigue trains humans to ignore real incidents; under-alerting lets users discover failures first.

Drift Detection

import numpy as np

def psi(reference, current, bins):
    p, _ = np.histogram(reference, bins=bins)
    q, _ = np.histogram(current, bins=bins)
    p = np.clip(p / p.sum(), 1e-6, 1)
    q = np.clip(q / q.sum(), 1e-6, 1)
    return float(np.sum((q - p) * np.log(q / p)))

def monitor(window, reference):
    metrics = {
        "p99_ms": np.quantile(window.latency_ms, .99),
        "error_rate": window.error.mean(),
        "toxic_rate": (window.p_toxic >= .8).mean(),
        "length_psi": psi(reference.char_count, window.char_count, [0,40,80,160,400,2000]),
        "missing_language": window.language.isna().mean(),
    }
    alerts = []
    if metrics["p99_ms"] > 200: alerts.append("serving_latency")
    if metrics["length_psi"] > .20: alerts.append("comment_length_drift")
    if metrics["missing_language"] > .02: alerts.append("language_schema")
    return metrics, alerts

Failure Catalogue

Silent failure

No event or outcome telemetry exists.

Missing logs

A broken logger makes the dashboard look quiet.

Wrong alert

Threshold ignores volume, seasonality, or slices.

Alert fatigue

Too many unactionable pages desensitise responders.

Dashboard overload

Hundreds of panels obscure the decision path.

Concept drift called a bug

Engineers roll back healthy infrastructure.

Latency-only monitoring

Fast harmful predictions look healthy.

Proxy-only monitoring

A proxy decouples from delayed true outcomes.

Interview Discussion

Can data drift occur without performance loss?
Yes. A changed feature may be irrelevant or remain inside a robust region. Drift triggers investigation, not automatic retraining.

How detect concept drift without immediate labels?
Use reports, appeals, review samples, disagreement and proxies, then confirm with delayed representative labels.

What belongs in an alert?
Impact, scope, threshold/window, evidence, owner, runbook, and recovery condition.

Incident Exercise And Review

Classify five signals into system, data, model, or business.
Distinguish data drift from concept drift with toxicity examples.
Calculate the three-bin PSI example without looking back.
Write a root-cause tree for rising moderator workload.
Design one page and two ticket alerts with windows and owners.
Tomorrow retrieve the four monitoring layers; in three days reproduce PSI and the investigation sequence.

What Comes Next?

Monitoring does not improve the model. It tells us when and why the system may need to change. Recall has fallen—but retraining immediately could amplify noisy labels or break stable slices. Next we design a safe update loop.