Where Do Ten Million Examples Come From?
Nobody manually writes ten million toxic comments for a training set. The examples already flow through the product: comments, reports, moderator queues, removals, and appeals. The engineering problem is deciding what to log, retain, sample, protect, and label.
A dataset is not a neutral snapshot of reality. It is the output of a collection system.
Lesson 1 specified the toxicity decision. That specification now tells us which events and context are relevant—and which information would be unavailable or improper to use.
The Data Supply Chain
Production data passes through choices: which users are represented, which fields are logged, which records survive privacy filters, which examples are sampled, and which receive labels. Each choice changes what the model can learn.
Learning objectives
- Design event logging and lineage
- Compare sampling strategies
- Measure coverage and long tails
- Detect collection bias and leakage
- Protect privacy and retention
- Use active learning and version datasets
Build The Data Pipeline
Click the data supply chain
Incoming comments: Raw text arrives with language, video, author, and time context.
Sampling lab
True deployment prevalence
The dataset now appears 10% toxic. The frequency is honest, but rare severe categories may have too few examples to learn or evaluate.
Overrepresented
Adequate
Undercovered
Long tail
Overrepresented
Policy-sensitive
Sources Are Signals, Not Ground Truth
| Source | Value | Bias or limitation |
|---|---|---|
| Historical comments | Natural language at production scale | Old policy and product behaviour may not match launch |
| Moderator actions | High-value expert decisions | Only content selected for review; action is not infallible truth |
| User reports | Finds harm the current system misses | Reporting rates differ by community and coordinated reports can be abused |
| Appeals | Rich false-positive signal | Only removed comments from users who appeal are visible |
| Public datasets | Fast baseline and external coverage | Different platforms, policies, populations, and licenses |
| Synthetic data | Targets rare or adversarial patterns | May reproduce generator artifacts rather than real behaviour |
Log Events That Can Be Reconstructed
A raw comment event needs a stable event ID, event time, policy version, text or a privacy-safe reference, language, conversation/video identifiers, and provenance for every later transformation. Processing time must remain separate from event time.
├── event_id, event_time
├── text_ref, language, video_category
├── reply_parent_id
├── consent_region, retention_class
└── schema_version
Without production logging, a team cannot recreate yesterday’s training data, investigate an incident, measure delay to moderation, or determine whether a policy change altered the data.
Distribution And The Long Tail
Imagine 90% of eligible comments are non-toxic, 9% are moderately toxic, and 1% are severe. A million-example random sample contains many severe cases in aggregate, but particular languages, threat types, and coded phrases may still have only a handful.
Random sampling preserves prevalence. Stratified sampling guarantees coverage across important slices. Importance or targeted sampling spends annotation budget on rare, costly, or uncertain cases. The training set may combine all three—but must retain each example’s sampling probability.
A balanced 50/50 dataset can help a learner see positives, yet it lies about deployment prevalence. It can distort calibration, accuracy, and workload estimates unless evaluation uses deployment-like data or correct weights.
Observed Data Versus The Deployment World
The collected dataset defines an empirical distribution:
Training works well when represents the deployment distribution for the decision we care about. Collection and sampling instead produce:
where is the chance an event is logged, retained, selected, and labelled. If that probability is known, importance weighting can correct some estimates:
Weights cannot recover groups that were never collected. Coverage comes before correction.
Collection Bias Is A Causal Story
Language bias
English is logged and reviewed more often than low-resource languages.
Creator bias
Large channels generate more events and moderator attention.
Reporting bias
Some communities report aggressively; others disengage silently.
Survivorship bias
Deleted accounts and removed threads disappear before collection.
Temporal bias
A policy or news event makes last year unlike next month.
Popularity bias
High-engagement videos dominate a naïve event sample.
Always ask: what process had to occur for this row to enter the table?
Leakage: Information From The Future
Suppose a training row includes moderator_removed=true or the later appeal outcome. Those fields almost reveal the label, but they do not exist when a fresh comment arrives. Randomly splitting duplicate comments lets near-identical text appear in train and test. Both create unrealistic evaluation.
- Use event-time joins: only features known at prediction time.
- Deduplicate before splitting, including near-duplicates and quote spam.
- Group related comments by thread, creator, campaign, or user where appropriate.
- Freeze the final test set before repeated modelling decisions.
Privacy Is An Architectural Constraint
Comments may contain names, contact details, health information, or location. User history creates a larger privacy surface. Collect only what the specification requires; separate identifiers from content; restrict access; encrypt data; record consent and legal basis; set deletion and retention schedules; and propagate deletion requests into derived datasets.
Hashing is not anonymisation when a stable hash can still link a person’s activity. Privacy decisions must be reviewed with policy and legal teams, especially across jurisdictions.
Feedback Loops Change Future Data
↓
model learns from reported comments
↓
model removes similar comments sooner
↓
users report different content—or adversaries change spelling
↓
future training distribution changes
The current system influences which labels will be observed. Preserve exploration samples and audit data outside the model-selected queue, or blind spots can reinforce themselves.
A Versioned Collection Pipeline
from dataclasses import dataclass
from hashlib import sha256
import pandas as pd
@dataclass(frozen=True)
class DatasetVersion:
name: str
policy_version: str
cutoff_time: str
sampling_seed: int
def canonical_hash(text: str) -> str:
normalised = " ".join(text.lower().split())
return sha256(normalised.encode()).hexdigest()
def build_sample(events: pd.DataFrame, version: DatasetVersion):
eligible = events[events.event_time <= version.cutoff_time].copy()
eligible["text_hash"] = eligible.text.map(canonical_hash)
eligible = eligible.drop_duplicates("text_hash")
sample = (eligible.groupby(["language", "toxicity_bucket"], group_keys=False)
.sample(n=500, replace=True, random_state=version.sampling_seed))
return sample.assign(dataset_version=version.name)Interview Discussion
Why keep a random holdout when using active learning?
An uncertainty-selected set is useful for improvement but cannot estimate ordinary deployment performance.
Why store sampling probabilities?
They make the selection process auditable and allow weighted estimates of deployment metrics.
Can more data fix collection bias?
More rows from the same biased pipeline reduce variance while preserving the bias.
Active Recall And Practice
- Why is a user report a useful but biased source?
- Compare random, stratified, and targeted sampling.
- Explain why 50/50 training prevalence can mislead production planning.
- Name two examples of future-information leakage.
- What metadata is required to reproduce a dataset?
- Design one audit slice for cross-language coverage.
What Comes Next?
Models learn from observed data, not reality. We have a traceable supply of comments, but no comment arrives with an unquestionable answer attached. Next we design the policy and annotation process that create labels.