Machine Learning from Scratch

Two Engineers, “The Same” Code

Engineer A

92.4% F1

Engineer B

89.8% F1

One used dataset v3 and seed 42. The other pulled the latest table, inherited seed 7, and had a newer tokenizer package. Neither run recorded those facts.

Can we trust either model if we cannot explain how it was produced?

A training pipeline turns model creation from an artisanal notebook session into a versioned, testable manufacturing process.

Retrieval Warm-Up

What must an online/offline feature parity test compare?
Why is a moderator decision after posting an illegal feature?
What four pieces define a rolling feature window?

Reveal after attempting

Same event, timestamp, feature versions, and values; it comes from the future; entity, clock, boundaries, and default/lateness policy.

Learning objectives

Make runs reproducible and deterministic
Move configuration out of code
Version data, features, code, environment, and models
Track experiments and lineage
Orchestrate stages as a recoverable DAG
Gate and register deployment candidates

Reproducibility And Gating Lab

Click the reproducible DAG

→

Load output: dataset manifest + checksums

Reproduce—or change—the run

Every changed input creates a new run identity.

92.4% F1

DatasetSeedLearning rate

Run	Data	Seed	LR	F1	P95 ms	Status
run-184	tox-v3	42	2e-4	0.924	38	candidate
run-185	tox-v4	42	2e-4	0.898	39	failed fairness
run-186	tox-v3	7	2e-4	0.917	38	tracked
run-187	tox-v3	42	5e-4	0.906	36	tracked

Quality gate

F1 ≥ .92

pass

Latency gate

P95 ≤ 50 ms

pass

Fairness gate

slice recall ≥ .80

pass

Security gate

signed artifact

pass

Every Stage Produces An Artifact

Stage	Input	Durable output
Load	dataset URI + version	manifest + checksums
Validate	raw snapshot	schema/quality report
Features	validated data + feature version	feature snapshot
Split	entity/time policy + seed	split manifest
Train	config + environment	checkpoints + logs
Evaluate	model + frozen eval sets	metrics, slices, plots
Register	signed candidate + evidence	immutable model version
Package	model + serving contract	container/image digest

An artifact is not just the final weights. A split manifest can explain leakage; a validation report can explain a changed metric; a container digest can recreate the runtime.

Configuration Is Data

Values hidden in notebook cells or source branches cannot be compared reliably. Put them in a validated configuration that is stored with the run:

run_name: toxicity-transformer-v18
dataset_version: tox-v3@sha256:8c4...
feature_version: tox-features-v4
model_architecture: distil-encoder-v2
learning_rate: 0.0002
batch_size: 256
epochs: 4
seed: 42
policy_version: toxicity-v1.0
environment_lock: uv.lock@sha256:dd1...

Changing any value creates a new run. Configuration validation should fail fast on missing fields, invalid ranges, incompatible feature/model versions, or an unrecognised policy.

What Reproducibility Means

Represent training algorithm $A$ as a function of dataset $D$ , hyperparameters $\theta$ , seed $s$ , code $c$ , and environment $e$ :

M=A(D,\theta,s,c,e)

Strict reproducibility asks the same inputs to produce the same artifact:

A(D,\theta,s,c,e)=A(D,\theta,s,c,e)

Some GPU operations remain nondeterministic. Then define a tolerance and record the distribution over repeat runs:

|m(M_1)-m(M_2)|\le\epsilon

Reproducible does not mean correct. It means differences are attributable rather than mysterious.

Randomness Has Multiple Sources

Initialisation

Weights begin at different points.

Shuffle and sampling

Mini-batches contain different examples and order.

Dropout and augmentation

Random masks or edits alter training signals.

Data-loader workers

Parallel completion changes ordering.

GPU kernels

Atomic operations may execute in nondeterministic order.

Distributed training

Reduction and retry timing can alter floating-point results.

Seed every library, request deterministic kernels when practical, record hardware, and repeat important experiments. A seed controls randomness; it does not remove uncertainty from a one-run conclusion.

Version The Whole Lineage

model toxicity-v18
├── dataset tox-v3 (manifest + row checksums)
├── labels policy-v1.0
├── features v4 / tokenizer toxic-2026-04
├── code commit 9f2c1a7
├── config run-184.yaml
├── environment image sha256:4aa…
└── parent checkpoint base-encoder-v2

Data-versioning systems such as DVC can map names to immutable snapshots and storage locations. The essential idea is content-addressed, immutable lineage—not a specific product.

Experiment Tracking Answers “What Changed?”

Each run records parameters, metrics, per-slice metrics, learning curves, confusion matrices, checkpoints, logs, notes, code commit, data and feature versions, timestamps, hardware, and owner. Comparison should change one deliberate variable where possible.

Worked diagnosis: run 184 scores .924 and run 185 scores .898. Code, seed, learning rate, and model match. Dataset changes from v3 to v4. Slice evaluation shows Spanish threat recall fell from .86 to .68; the v4 manifest reveals a deduplication rule removed repeated threat templates. The tracker turns “the model got worse” into an inspectable causal hypothesis.

Orchestration Is A Recoverable DAG

validate_data ─┬→ compute_features → split → train ─┬→ evaluate_quality ─┐
└→ privacy_check ├→ evaluate_slices ├→ gate → register
└→ benchmark_latency┘

Edges express artifact dependencies. Independent evaluations run in parallel. If latency benchmarking fails transiently, retry that node—not eight hours of training. Cache outputs only when all upstream content hashes match. Orchestrators schedule work, persist state, retry safely, and expose failure ownership.

Automated Gates Protect Production

Gate	Example rule	Why
Quality	F1 ≥ .92 and severe recall ≥ .95	Aggregate quality and costly failures
Fairness	Every language slice recall ≥ .80	Prevent one average hiding a harmed group
Latency	P95 model inference ≤ 50 ms	Fit the serving budget
Drift	Feature PSI below approved threshold	Catch unexpected dataset change
Security	Artifact signed; dependency scan passes	Protect supply chain
Regression	No critical policy case regresses	Preserve known behaviour

A pass creates a deployment candidate, not an automatic right to serve all traffic. High-impact changes may still require review and a staged rollout.

Registry, Stages, And Rollback

run artifact → registry/toxicity-v18
                 ├── candidate (immutable)
                 ├── staging alias
                 └── production alias → toxicity-v17

rollback = move production alias back to v17

The registry stores approved artifacts, schemas, metrics, signatures, lineage, and stage history. Aliases point to immutable versions. Never overwrite “model.pkl” in place; rollback requires the exact previous model and serving contract.

A Modular Training Pipeline

def pipeline(config):
    run = tracker.start(config=config, git_commit=git_sha(), started_at=utcnow())
    data = load_version(config.dataset_version)
    validate(data, schema="toxicity-events-v3")
    features = compute_features(data, config.feature_version)
    train, valid, test = split_from_manifest(features, config.split_manifest)
    model = train_model(train, config, seed=config.seed)
    metrics = evaluate(model, valid, slices=["language", "severity"])
    latency = benchmark(model, serving_shape=config.serving_shape)
    artifacts = tracker.log(run, metrics=metrics, latency=latency, model=model)
    gates.require(metrics["f1"] >= .92, metrics["severe_recall"] >= .95)
    return registry.register(artifacts, lineage=run.lineage)

# CI runs unit tests, a tiny end-to-end fixture, schema checks, and packaging.
# Scheduled/approved jobs run full training and deployment gates.

Failure Catalogue

Manual notebook

State, execution order, and outputs are hidden.

Missing config

Defaults vary by machine or engineer.

Untracked data

The same query returns different rows later.

Lost checkpoint

An expensive run cannot resume or be audited.

Accidental overwrite

Rollback and comparison become impossible.

Hidden preprocessing

Training lineage omits the effective model input.

Library drift

Serialization or numerical behaviour changes.

Silent partial failure

A stale artifact is mistaken for a fresh result.

Interview Discussion

What must be versioned besides weights?
Data, labels, features, code, config, dependencies, base model, evaluation sets, and serving contract.

How do CI and training orchestration differ?
CI quickly validates code and small fixtures; orchestration executes and recovers the full artifact DAG, often on scheduled compute.

Can exact reproducibility hurt performance?
Deterministic kernels may be slower. Choose the required tolerance, record nondeterminism, and repeat important comparisons.

Apply And Review

Write the complete input tuple for $M=A(D,\theta,s,c,e)$ .
Explain the difference between an experiment tracker and registry.
Draw a DAG where three evaluations run in parallel after training.
Design quality, fairness, latency, and security gates for toxicity-v18.
Diagnose the 92.4 versus 89.8 discrepancy using lineage, not guesses.
Tomorrow retrieve config, lineage, and registry; in three days reconstruct the DAG and rollback path.

What Comes Next?

A good training pipeline lets another engineer reproduce today’s model six months from now. We now have a signed, evaluated candidate. The next question is physical: where does inference happen when millions of users press Post?