Machine Learning from Scratch

Recall Fell. Train Now?

Monitoring shows toxicity recall fell from 91% to 78%, especially for new Spanish slang. Retraining tonight feels decisive. It may also be wrong: recent reports are selection-biased, labels are incomplete, and a rushed candidate could reduce appeals in one language while increasing false removals elsewhere.

Retraining is a production change, not a repair button.

It consumes labels and compute, changes system behaviour, and can regress cases the current model handles well. Retraining should be triggered by evidence and pass the same disciplined pipeline as the original model.

Retrieval Warm-Up

Why does PSI above 0.2 not prove model quality fell?
Which monitoring signal can confirm calibration drift?
What must a rollback alias point to?

Reveal after retrieving

Drift may be harmless; delayed representative outcomes compared with predicted bins; an immutable compatible model artifact and contract.

Learning objectives

Choose evidence-based triggers
Refresh data and labels safely
Balance freshness with forgetting
Backtest and replay candidates
Run regression, fairness, calibration, and latency gates
Version, stage, deploy, and roll back

Update Policy And Rollback Lab

Retraining policy simulator

Frequency reduces staleness but consumes labels and compute.

every 30 days

Training window: 180 days

Estimated freshness

83%

Runs / year

400

Forgetting risk

Candidate gate

Recall

91% → 94%

pass

Appeals

1.8% → 2.7%

fail

Spanish recall

82% → 90%

pass

P99 latency

142 → 149 ms

pass

Registry and rollback

Production alias → v18

The System Is A Loop

serve v18 → collect events/outcomes → monitor
↑ ↓
deploy safely ← gate/register ← retrain on approved Dₜ₊₁
└──────────────── monitor again ────────────────┘

The model influences moderation, which influences reports and appeals, which influence future labels. Preserve representative audit samples so the new dataset does not consist only of cases selected by the old model.

Why A Model Becomes Stale

New slang

Users invent coded insults absent from v18.

New populations

Language and community mix expands.

Policy change

A formerly allowed category becomes reviewable.

Adaptive spam

Attackers learn current model boundaries.

Product change

Replies, Shorts, or context fields alter inputs.

Label process change

Guidelines and annotator pool evolve.

First diagnose what changed. A policy change may require relabelling and new output semantics, not merely adding recent rows. A schema failure requires fixing the pipeline, not retraining around broken inputs.

Scheduled Versus Triggered

Policy	Best when	Risk
Weekly/monthly	Labels arrive steadily and drift is predictable	Wasteful runs or insufficient time for quality labels
Performance-triggered	Representative delayed metrics cross a material boundary	Detection arrives late and labels may be sparse
Drift-triggered	Input movement is known to threaten quality	Drift may be harmless; automatic retrain amplifies noise
Human-triggered	Policy/product incident requires judgement	Slow or inconsistent without a runbook
Hybrid	Cadence supplies baseline; evidence can accelerate or postpone	Requires explicit precedence and ownership

Concrete trigger: open a retraining review when seven-day severe recall is below 88% with at least 1,000 adjudicated examples, or a policy owner approves a version change. The review can decide not to train.

Freshness Is Not “Use Only Recent Data”

historical anchor ───────── recent weighted window ─ cutoff
known edge cases ───────── new slang + new languages ─┤
label delay →

A short 30-day window adapts quickly but forgets rare threats and seasonal behaviour. A multi-year window preserves coverage but dilutes emerging patterns. Combine a stable historical anchor, recent representative sample, targeted failures, and immutable golden cases. Keep post-cutoff events out of training and evaluation.

Labels need time to mature. Training on immediate user reports mistakes “reported” for “toxic” and excludes later appeals.

A Sequence Of Versioned Learners

At time $t$ , training algorithm $A$ consumes approved dataset $D_t$ :

M_t=A(D_t,\theta_t,s_t,c_t,e_t)

The candidate is:

M_{t+1}=A(D_{t+1},\theta_{t+1},s_{t+1},c_{t+1},e_{t+1})

Promotion is constrained improvement, not one metric maximisation:

\max\ \Delta U\quad\text{s.t.}\quad \Delta R_{severe}\ge0,\ \Delta R_{slice}\ge-\epsilon,\ P99\le150ms,\ \text{appeal rate}\le g

$U$ is product utility; the constraints encode regression and operational guardrails.

What May Change?

policy/label schema ─┬→ dataset version ─┬→ feature version ─→ model
new raw events ──────┘ ├→ split manifests
code/config/env ─────────────────────────┘
model architecture/config ──────────────────────────┘

Change one conceptual layer at a time where possible. If policy, data, tokenizer, and architecture all change together, a win or regression is difficult to attribute. A policy-version change may require parallel evaluation rather than comparison against old labels.

Evaluate Before Any Live Decision

Evidence	Comparison
Frozen test and golden cases	v18 versus candidate under identical policy labels
Recent representative test	Current language, slang, and prevalence
Slices	Language, severity, creator size, context availability
Calibration	Reliability bins and threshold workload
Operational benchmark	P50/P95/P99, memory, throughput, contract compatibility
Business proxies	Reports, simulated queue, expected appeals—not causal proof

Replay And Shadow The Candidate

Historical replay: send time-ordered, point-in-time-correct events through v18 and v19. Measure disagreement, queue volume, latency, and known outcomes. Replay catches contract and state bugs that a static dataframe can hide.

Shadow serving: copy current live requests to v19, log outputs, but let v18 decide. Shadow reveals production latency, missing features, and prediction shifts. It still cannot measure how users react to v19.

Inspect disagreements, especially v18=keep, v19=remove and high-confidence reversals. A small aggregate difference can concentrate in a high-risk slice.

Regression Tests Preserve Institutional Memory

A golden suite contains fixed, adjudicated examples for threats, quotes, reclaimed terms, adversarial spelling, each supported language, and previous incidents. It is versioned with policy.

case incident-2026-014
input: quote containing a threat in documentary context
required action: review (not automatic removal)
policy: toxicity-v1.0
reason: context-dependent quotation
candidate v19: REMOVE → regression gate FAIL

Golden tests protect known behaviour but are not a representative performance estimate. Use both golden and sampled test sets.

Rollback Is A Tested Operation

Keep v18’s signed artifact, feature contract, container, thresholds, and capacity warm during rollout. Rollback criteria might include P99 > 180 ms, error rate > 1%, appeal guardrail breach, or a critical policy incident.

Moving the production alias back is safe only if data/schema changes remain backward compatible. Database migrations and new feature requirements can make model rollback impossible unless designed jointly.

Evidence-Gated Retraining

def retrain(trigger, config):
    assert trigger.has_evidence and trigger.owner_approved
    dataset = build_dataset(
        historical_anchor="tox-anchor-v5",
        recent_cutoff=config.label_maturity_cutoff,
        targeted_slices=trigger.affected_slices,
    )
    candidate = training_pipeline.run(dataset=dataset, config=config)
    report = compare(
        incumbent=registry.load("production"), candidate=candidate,
        suites=["frozen-test-v7", "recent-representative-v3", "golden-policy-v12"],
        checks=["quality", "calibration", "fairness", "latency", "contract"],
    )
    gates.require_all(report)
    registry.register(candidate, lineage=dataset.lineage, report=report)
    replay(candidate, traffic="last-14-days", point_in_time=True)
    shadow(candidate, duration="48h")
    return candidate  # still not full production; A/B test next

Failure Catalogue

Noisy labels

Recent reports are treated as truth before adjudication.

Contamination

Training rows leak into golden or recent test sets.

Policy drift

Old and new label meanings are mixed.

Recent overfit

Candidate learns one incident and hurts ordinary traffic.

Catastrophic forgetting

Rare historical threats disappear from a short window.

No comparison

Candidate is judged alone, not against v18.

No rollback

Incompatible schemas make reversal impossible.

Feedback amplification

Old model-selected cases dominate new learning.

Interview Discussion

Should drift automatically retrain?
Usually no. Validate impact and cause; otherwise the system may train around bugs or harmless shifts.

How prevent catastrophic forgetting?
Retain historical anchors and golden cases, sample rare categories, and enforce old-slice regression gates.

What does shadow serving prove?
Contract, latency, and output behaviour on live inputs—not causal product improvement.

Candidate Review And Spaced Practice

Turn “recall fell” into a precise retraining-review trigger.
Design a dataset mixing history, recent representative data, and targeted failures.
Explain the distinct evidence from frozen, recent, and golden sets.
Write promotion and rollback gates for v19.
Explain why replay must be point-in-time correct.
Tomorrow retrieve triggers and dataset composition; in three days reproduce the complete candidate path.

What Comes Next?

Retraining should be triggered by evidence, not habit. Candidate v19 passes offline tests, replay, and shadow serving. That still does not prove it improves real conversations. The final lesson measures its causal product impact.