Feature Pipelines

The Model Did Not Change. Accuracy Collapsed.

Retrieval Warm-Up

1. Why must a feature exist at prediction time?
2. Which lesson artifact tells us what “toxicity” means?
3. Why can a balanced annotation set distort deployment estimates?

Learning objectives

• Separate raw observations from model features
• Eliminate training-serving skew
• Design offline, online, batch, and streaming features
• Compute point-in-time-correct stateful features
• Understand feature stores and lineage
• Validate schemas, values, and distributions

Feature Pipeline Failure Lab

One Comment, Seven Representations

The Model Sees Features, Not Reality

Offline And Online Must Share A Definition

The storage and execution engines may differ. The feature definition, schema, defaults, tokenizer, windows, and versions must not. Prefer one shared transformation library or declarative feature definition over copied notebook and service code.

Point-In-Time Correctness Prevents Leakage

Stateful Features Need Window Semantics

What A Feature Store Solves

Validate Before The Model Receives Anything

One Deterministic Pipeline For Both Paths

from dataclasses import dataclass
import unicodedata

@dataclass(frozen=True)
class FeatureConfig:
    version: str = "tox-features-v4"
    tokenizer_version: str = "toxic-2026-04"
    unknown_language: str = "und"

def normalise(text: str) -> str:
    text = unicodedata.normalize("NFKC", text)
    return " ".join(text.split()).casefold()

def transform(event, tokenizer, language_detector, config=FeatureConfig()):
    text = normalise(event["text"])
    language = language_detector(text) or config.unknown_language
    token_ids = tokenizer.encode(text)
    return {"token_ids": token_ids, "language": language,
            "char_count": len(text), "feature_version": config.version}

# Training and serving import this exact function.
assert transform(sample, tok, detect) == transform(sample, tok, detect)

Miniature timestamped store

class FeatureStore:
    def __init__(self): self.values = {}

    def write(self, entity, feature, event_time, value):
        self.values.setdefault((entity, feature), []).append((event_time, value))

    def lookup(self, entity, feature, as_of):
        valid = [(t, v) for t, v in self.values.get((entity, feature), []) if t <= as_of]
        return max(valid)[1] if valid else 0

    def offline_export(self, entities, feature, timestamps):
        return [self.lookup(e, feature, t) for e, t in zip(entities, timestamps)]

Failure Catalogue

Interview Discussion

Apply And Review

1. For one comment, name every representation before prediction.
2. Explain training-serving skew without using the word “different.”
3. Decide whether “reports in the next hour” is legal and justify using the event timeline.
4. Specify a 30-day user feature with entity, clock, boundary, and default.
5. Design three parity tests and three production validations.
6. In one day, retrieve skew, point-in-time correctness, and feature lineage; in three days, reproduce the stateful-feature formula without notes.

What Comes Next?

Models rarely fail because matrix multiplication changes. They fail because inputs quietly change. With reproducible inputs, the next systems question is how another engineer can recreate the entire trained model six months from now.