Machine Learning from Scratch

96% Offline—Ship It?

Model A · production

94%

Model B · candidate

96%

B may remove abuse faster. It may also increase wrongful removals, appeals, moderator work, latency, or creator churn. Offline examples cannot reveal every behavioural response.

Offline metrics estimate prediction quality. A controlled experiment estimates product impact.

Final Retrieval Warm-Up

What can shadow serving prove, and what can it not prove?
Why must v19 retain golden cases and a historical anchor?
What would cause immediate rollback during a canary?

Reveal after retrieving

Live contract/latency/output behaviour, not user impact; prevent forgetting known failures; predeclared safety, reliability, or policy guardrail breach.

Learning objectives

Randomise users into control and treatment
Choose one primary metric and guardrails
Interpret effects, confidence intervals, p-values, and power
Choose allocation and duration
Avoid peeking, interference, and biased assignment
Make a launch decision from total evidence

Experiment And Lifecycle Lab

Experiment simulator

Outcome: percentage-point change in appeal rate (lower is better).

-0.35 pp

Total users: 20,000Traffic to B: 50%True effect: -0.35 ppOutcome noise: 1.2

95% CI

[-0.38, -0.32]

Power

78%

Conclusion

detectable

Split

50/50

Live decision dashboard

Primary: reports

-6.1%

win

Appeals

+14.2%

fail

P99 latency

+5 ms

pass

Creator retention

+0.1%

neutral

Decision: do not ship. The primary metric improved, but a predeclared appeal-rate guardrail failed.

The complete feedback loop

Problem→

Data→

Labels→

Features→

Training→

Serving→

Monitoring→

Retraining→

A/B test

↺

The Experiment Architecture

eligible users / creator clusters
             ↓ deterministic random assignment
     ┌───────┴────────┐
control A: v18    treatment B: v19
     └───────┬────────┘
   join exposure + outcomes
             ↓
primary metric + guardrails + slices → ship / iterate / stop

Log assignment before outcome, exposure after the request actually uses the assigned model, and analyse by assigned group. Stable hashing keeps a user in one group across sessions.

Randomisation Builds A Fair Counterfactual

We cannot show the same user at the same moment both v18 and v19. Random assignment creates groups that are comparable in expectation, so their outcome difference estimates what v19 caused.

Selection-bias failure: route English desktop users to v19 and everyone else to v18. If appeals differ, model version is tangled with language and device. No statistical formula repairs that design.

Before launch, run an A/A test or sample-ratio-mismatch check: expected 50/50 assignment should not produce 57/43, and baseline covariates should balance within chance.

Predeclare Metrics And Decision Rules

Role	Metric	Example rule
Primary	User reports per 1,000 exposed comments	Ship if meaningfully lower
Guardrail	Successful appeal rate	Do not increase more than 0.2 pp
Guardrail	Severe-toxicity time-to-removal	Must not worsen
Guardrail	P99 decision latency / availability	≤150 ms / ≥99.95%
Secondary	Moderator minutes per 1,000 comments	Explain operational cost
Long-term	Creator 28-day retention	Watch for delayed harm

Choose metrics, minimum detectable effect, duration, slices, exclusions, and stopping rules before looking at results. Changing the primary metric after seeing data converts noise into a story.

Effect And Uncertainty

For a numeric outcome, the observed difference is:

\hat\Delta=\bar Y_B-\bar Y_A

With independent groups, an approximate standard error is:

SE(\hat\Delta)=\sqrt{\frac{s_A^2}{n_A}+\frac{s_B^2}{n_B}}

A 95% confidence interval is:

\hat\Delta\pm1.96\,SE(\hat\Delta)

Worked example. Reports per 1,000 comments are 12.0 for A and 11.4 for B, so $\hat\Delta=-0.6$ . If $SE=0.2$ , the interval is:

-0.6\pm1.96(.2)=[-0.992,-0.208]

The interval excludes zero and all plausible effects reduce reports. Statistical significance asks whether chance variation under a null effect could readily produce the observation. The p-value is that tail probability under the null—not the probability the model is good.

Power, Sample Size, And Duration

Power is the probability of detecting the minimum effect we care about when it truly exists. It rises with sample size and effect size, and falls with outcome variance and extreme allocation.

A 50/50 split yields maximum statistical information for a fixed sample. A 90/10 or 99/1 rollout limits treatment risk but needs more total traffic for the same precision. Run long enough to reach planned sample size and cover weekly cycles and outcome delay.

Too short produces noisy wins and misses delayed appeals. Too long wastes opportunity and exposes users to a losing treatment. Duration is computed from power and operational cycles, not “until p < .05.”

Peeking Creates False Discoveries

Repeatedly checking an ordinary fixed-horizon p-value and stopping at the first significant moment increases false positives. Random fluctuation gets many chances to look convincing.

Use the predeclared fixed sample/duration, or a valid sequential method with adjusted boundaries or always-valid inference. Safety guardrails can still stop an experiment immediately; safety stopping and efficacy claiming are different decisions.

Canary And A/B Answer Different Questions

Shadow

Can it run safely without affecting users?

Canary

Does a small live exposure remain operationally safe?

A/B test

What causal product effect does v19 have?

A sensible sequence is shadow → 1% canary → powered randomised experiment → progressive rollout → continued monitoring.

Interference Breaks User-Level Independence

One moderation decision changes a public reply thread seen by control and treatment users. Creators may change behaviour after removals. Outcomes for one unit therefore depend on another unit’s treatment.

Randomise at a cluster that contains the interaction—such as creator/channel or conversation thread—when spillovers are material. Then calculate uncertainty using clusters, not individual comments. Cluster randomisation reduces effective sample size because outcomes within a cluster resemble one another.

Potential Outcomes Intuition

For unit $i$ , imagine $Y_i(1)$ under v19 and $Y_i(0)$ under v18. The individual effect is unobservable because we see only one:

ATE=\mathbb E[Y(1)-Y(0)]

Randomisation makes the observed group difference an unbiased estimate of this average treatment effect under correct assignment, exposure, and no problematic interference.

The Launch Review

Example result

Primary: user reports −6.1%, 95% CI [−8.4%, −3.8%]
Appeals: +14.2%, guardrail limit +5% → FAIL
P99 latency: +5 ms, within 150 ms SLO
Severe recall proxy: +3.0%
Spanish appeal increase: +24%, concentrated regression

Decision: do not fully ship. The candidate reduces reports but removes too much legitimate Spanish content. Investigate thresholds, labels, and slice calibration; build v19.1 and repeat the evidence path.

Simulate And Analyse Assignment

import hashlib, numpy as np

def assignment(cluster_id: str, experiment="tox-v19", treatment_share=.5):
    value = int(hashlib.sha256(f"{experiment}:{cluster_id}".encode()).hexdigest()[:8], 16)
    return "B" if value / 0xffffffff < treatment_share else "A"

def difference_in_means(outcomes, groups):
    a, b = outcomes[groups == "A"], outcomes[groups == "B"]
    effect = b.mean() - a.mean()
    se = np.sqrt(b.var(ddof=1)/len(b) + a.var(ddof=1)/len(a))
    return {"effect": effect, "ci95": (effect-1.96*se, effect+1.96*se)}

# Analyse by assigned group (intention-to-treat), verify sample ratio,
# report the predeclared primary metric, every guardrail, and slices.
result = difference_in_means(report_rate, assigned_group)

Interview Discussion

Why randomise by creator instead of comment?

Moderation changes a shared community; cluster assignment reduces contamination across comments and viewers.

What if the primary metric wins but a guardrail fails?

Follow the predeclared rule: stop or withhold launch, diagnose, and iterate.

What does a confidence interval communicate?

A range of effect sizes compatible with the estimator and assumptions, showing both direction and precision.

Final Capstone

1. Define eligibility, randomisation unit, control, treatment, exposure, and duration.

2. Choose one primary metric, three guardrails, and minimum detectable effects.

3. Compute a difference and 95% interval from provided means and standard error.

4. Explain why peeking and interference threaten the conclusion.

5. Make a ship/iterate/stop decision from the launch-review evidence.

6. From memory, narrate one comment from Post → features → inference → telemetry → label → retraining → experiment.

Spaced review

Tomorrow: randomisation, primary/guardrail roles, confidence interval. Three days: experiment failure modes. One week: redraw and explain the entire nine-lesson feedback loop without notes.

Failure Catalogue

Stopping early: A random fluctuation becomes a launch.
Metric switching: The team promotes whichever measure happened to win.
Unequal groups: Assignment or exposure bug creates selection bias.
Seasonality: Weekday-only data fails to represent weekends or events.
Novelty effect: Users react temporarily to changed moderation.
Simpson’s paradox: Aggregate win hides a loss caused by group composition.
Ignored guardrail: Reports fall while wrongful removals surge.
Offline triumphalism: Accuracy is mistaken for causal product value.

The ML Systems Lifecycle

A machine learning model is not a product.

A production ML system is a continuously evolving feedback loop connecting users, data, labels, features, models, infrastructure, decisions, outcomes, and engineers.

That is why successful ML teams spend as much effort on systems as they do on algorithms.