Machine Learning from Scratch

A Decision In 150 Milliseconds

User presses Post: “You're an idiot.”

The user expects the comment to appear immediately. Before publication, YouTube may authenticate the request, retrieve fresh features, run the model, apply thresholds, record the decision, and return a response. The product budget is 150 ms.

Where does inference actually happen—and what happens when that machine is slow or unavailable?

A saved model file is not a product. Serving is the system that delivers reliable predictions under traffic, latency, and failure constraints.

Retrieval Warm-Up

What immutable information must a registry entry contain?
Why does passing an offline quality gate not justify instant 100% rollout?
What should happen if only the latency DAG node fails?

Reveal after retrieving

Artifact, contract, signature, metrics, and lineage; live systems add new risks; retry the failed node while reusing valid cached parents.

Learning objectives

Trace online and offline inference
Budget latency and tail latency
Reason about throughput, queues, and batching
Autoscale and cache safely
Design fallbacks and graceful degradation
Use shadow, canary, SLO, and rollback practices

Live Serving Simulator

Trace one 150 ms request

API

8 ms

→

Load balancer

3 ms

→

Feature service

31 ms

→

Inference

47 ms

→

Decision

4 ms

→

Database

22 ms

Traffic, batching, and autoscaling

Increase load and keep tail latency inside the SLO.

P99 90 ms

Traffic: 700 RPSReplicas: 4Batch size: 4

Capacity

1192 RPS

Utilisation

59%

Queue

P50

50 ms

Failure drill

Fallback: primary model

Canary traffic

5% new

Old model: 95% · New model: 5%. Promote only if quality, latency, errors, appeals, and slices remain healthy.

Trace One Request, Millisecond By Millisecond

Elapsed	Subsystem	Work
0–8 ms	Edge/API	Authenticate, validate schema, assign request ID
8–11 ms	Load balancer	Route to a healthy, compatible replica
11–42 ms	Feature service	Text features plus point-in-time user history
42–89 ms	Inference server	Batch, tensorise, execute toxicity-v18
89–93 ms	Decision policy	Apply keep/review/remove thresholds
93–115 ms	Database/event log	Persist decision, versions, latency, and fallback state
115–128 ms	Response	Serialize and return product action

The remaining 22 ms is safety margin for normal variance. Request IDs connect traces across services; model, feature, policy, and API versions make every decision auditable.

Latency, Throughput, And Availability

End-to-end latency is a critical-path sum:

T=T_{network}+T_{features}+T_{queue}+T_{inference}+T_{post}

Throughput is completed requests per second:

R=\frac{N_{completed}}{\Delta t}

If arrival rate $\lambda$ approaches service capacity $\mu$ , queues grow sharply. For a simple M/M/1 intuition:

\mathbb E[T]=\frac{1}{\mu-\lambda},\qquad \lambda<\mu

Availability over a window is:

A=\frac{\text{successful service time}}{\text{total scheduled time}}

These quantities trade off. Larger batches increase hardware throughput but wait longer to fill; extra fallbacks improve availability but may reduce quality.

Averages Hide The Users Who Wait

Sort request latencies. P50 is the median, P95 is slower than 95% of requests, and P99 captures the slowest one percent. If one billion comments are served, one percent is ten million experiences.

P50

62 ms

P95

118 ms

P99

147 ms

Mean

71 ms

A 150 ms P99 SLO says at least 99% of eligible requests should complete within 150 ms over a defined window. Pair latency with availability and quality; a service can meet latency by returning errors instantly.

Online Or Batch?

Online inference

One live comment needs an immediate moderation action. Fresh inputs, low latency, high availability, smaller batches.

Batch inference

Rescore a historical corpus or prepare monthly recommendations. High throughput, relaxed latency, large efficient batches.

A hybrid system can run cheap online screening and deeper asynchronous review. The product action determines the appropriate path.

Dynamic Batching Trades Waiting For Efficiency

A GPU processes a tensor batch efficiently. Serving one comment per execution wastes parallel capacity. A dynamic batcher waits up to, say, 4 ms or until 16 requests arrive, whichever happens first.

requests arrive: A · B C · D E F G
└── max_wait=4ms / max_batch=4
GPU batches: [A,B,C] [D,E,F,G]

At high traffic, batches fill quickly and both throughput and latency improve. At low traffic, waiting can dominate. Benchmark batch size against real request shapes and tail latency—not only GPU utilisation.

Autoscaling Needs Headroom

Scale replicas from queue depth, concurrency, RPS, or hardware utilisation. Scaling only after P99 breaks is too late because new replicas need time to start, load weights, and warm caches.

Use minimum warm capacity, predictive scaling for known peaks, maximum limits to control cost, and scale-down cooldowns to avoid oscillation. Load balancers should route only to healthy replicas with the correct model and feature contract.

Cold start: container scheduling + image pull + model download + GPU allocation + weight loading + compilation/warm-up. Large models can make this minutes, not milliseconds.

Graceful Degradation Is Designed In Advance

Failure	Fallback	Product consequence
Primary model timeout	Older compatible model	Slight quality loss, preserve speed
Feature service down	Text-only model	Lose user-history signal; widen review band
GPU unavailable	CPU rules + queue uncertain cases	Lower recall, higher moderator load
All inference unavailable	Publish low-risk, hold high-risk	Availability with delayed moderation
Database degraded	Durable event queue	Respond safely, persist asynchronously

Every response should record whether it came from the primary model, fallback, cache, or human path. Otherwise aggregate metrics silently mix different systems.

Shadow Before Canary; Canary Before Full Rollout

Shadow

Copy live requests to toxicity-v18. Its predictions affect no users. Compare latency, errors, prediction distributions, and disagreement with v17.

Canary

Route 1–5% of eligible traffic to v18. Its decisions are real. Monitor technical and product guardrails before increasing exposure.

Shadowing cannot measure behavioural consequences because users never see the shadow action. Canary data can, but creates real risk. Roll out 1% → 5% → 25% → 50% → 100% with explicit hold periods and automatic rollback thresholds.

FastAPI Serving Contract

from fastapi import FastAPI
from pydantic import BaseModel
from time import perf_counter

app = FastAPI()
model = registry.load("toxicity-v18")
features = FeatureClient(version="tox-features-v4")

class Comment(BaseModel):
    request_id: str
    text: str
    user_id: str
    event_time: str

@app.post("/predict")
def predict(comment: Comment):
    start = perf_counter()
    z = features.for_event(comment.model_dump())
    probability = float(model.predict_proba(z))
    decision = "remove" if probability >= .80 else "review" if probability >= .35 else "keep"
    return {"probability": probability, "decision": decision,
            "model_version": "toxicity-v18", "feature_version": "tox-features-v4",
            "latency_ms": (perf_counter() - start) * 1000}

Bounded dynamic batch

async def batch_worker(queue, max_batch=16, max_wait_ms=4):
    while True:
        first = await queue.get()
        batch = [first]
        deadline = loop.time() + max_wait_ms / 1000
        while len(batch) < max_batch and loop.time() < deadline:
            try: batch.append(await asyncio.wait_for(queue.get(), deadline-loop.time()))
            except asyncio.TimeoutError: break
        outputs = model.predict_proba([item.features for item in batch])
        for item, output in zip(batch, outputs): item.future.set_result(output)

Production Dashboard

Traffic

RPS, concurrency, queue depth, batch size

Latency

P50/P95/P99 end-to-end and by component

Resources

GPU/CPU utilisation, memory, replica count

Reliability

Error, timeout, retry, fallback, and availability

Predictions

Score/decision distribution by model and slice

Rollout

Shadow disagreement, canary exposure, rollback state

Correlate panels using model, feature, policy, region, replica, and request versions. A latency spike isolated to one model version suggests a different response than a platform-wide network spike.

Failure Catalogue

Model timeout

Bound work, cancel, fallback, and preserve the latency budget.

Feature outage

Use compatible defaults or degraded model; never silently inject stale state.

Memory leak

Replica degrades over hours and restarts; watch memory slope.

Traffic spike

Queues grow faster than cold replicas can start.

API incompatibility

New tensor shape or field breaks old clients.

Cold start

First requests pay model load and compilation cost.

Version mismatch

Model v18 receives tokenizer or feature schema v3.

Prediction drift

Score distribution shifts even while service health looks green.

Interview Discussion

Why can batching increase both throughput and latency?
It uses parallel hardware better but each request may wait for companions.

Why monitor P99 instead of only mean?
Queueing, cold starts, and stragglers harm a large absolute number of users while barely moving the mean.

When is a shadow deployment insufficient?
When the outcome depends on user or moderator reactions to the new decision; shadow predictions do not change behaviour.

Outage Drill And Review

Allocate a concrete 150 ms latency budget across six subsystems.
Explain what happens as $\lambda\to\mu$ .
Choose batch size and maximum wait for low versus peak traffic.
Design a safe feature-service fallback and state its quality cost.
Write shadow, 5% canary, promotion, and rollback criteria.
Tomorrow retrieve latency/throughput/availability; in three days redraw the full request path and failure fallbacks.

What Comes Next?

A model is useful only if it delivers reliable predictions inside the product’s latency budget. The service is live. Now we need to know when latency, inputs, predictions, labels, or user outcomes begin to deteriorate: monitoring.