Machine Learning from Scratch

Where Does The Correct Answer Come From?

“You're absolutely brilliant.
Nice job ruining everything.”

Toxic or not toxic? Reasonable people can disagree. The words look positive; the pragmatic meaning is hostile; whether it violates policy may depend on the reply chain and whether it targets a person.

A label is not discovered in the row. It is produced by people applying a policy with limited context.

Ground Truth Is Often A Process

For toxicity, “ground truth” means the best defensible policy judgement available under a documented annotation process. It may combine independent annotators, expert adjudication, confidence, context, and a policy version.

Learning objectives

Design a label schema and guidelines
Measure inter-annotator agreement
Represent ambiguity with soft labels
Distinguish random and systematic noise
Use active learning and weak supervision
Monitor policy and label drift

Annotation And Agreement Lab

Annotate before revealing the group

“You're absolutely brilliant. Nice job ruining everything.”

Ten-annotator vote lab

Change how many annotators vote toxic.

5 / 10

Majority label

Toxic

Soft label

0.50

Confidence

A hard majority collapses 6–4 and 10–0 into the same target. A soft label preserves how much humans agreed.

Agreement snapshot

Observed agreement

0.71

Chance agreement

0.50

Cohen-style κ

0.43

Design The Concept Before The Interface

A binary toxic / not toxic label is inexpensive and easy to train, but combines insults, threats, hate speech, and harassment. A multilabel schema supports different actions and audits. A hierarchy can first ask whether policy is violated, then identify type and severity.

Label	Operational definition	Example
Insult	Attack on a person or group’s attributes or competence	‘Only an idiot believes that.’
Threat	Credible or aspirational intent to cause harm	‘I will find you after the stream.’
Hate speech	Attack based on a protected characteristic	Dehumanising a religious group
Harassment	Repeated or targeted unwanted hostility	Following a creator across videos to abuse them
Spam	Manipulative, repetitive, or deceptive promotion	Repeated fake giveaway links

Schema granularity trades annotation reliability and cost against downstream control. If annotators cannot apply a distinction consistently, the model will not learn it reliably.

Annotation Guidelines Are Executable Policy

Example: targeted insult

Label insult when negative language attacks a person or group rather than an idea.
Do not label ordinary disagreement, criticism of an argument, or non-targeted profanity.
Use the reply chain to resolve pronouns and quoted text.
If the target or intent cannot be resolved, choose needs context; do not guess.
Also mark severity and whether a protected characteristic is targeted.

Positive: “You are a pathetic liar.”

Negative: “This argument is badly supported.”

Borderline: “Brilliant work, genius”—review the conversational context.

Guidelines need definitions, positive and negative examples, borderline cases, context rules, an abstain option, and escalation criteria. Pilot them before scaling annotation.

Inter-Annotator Agreement

Raw agreement can look impressive when one label dominates. Cohen’s kappa compares observed agreement $P_o$ to the agreement expected from the annotators’ marginal label frequencies $P_e$ :

\kappa=\frac{P_o-P_e}{1-P_e}

Worked example. Two annotators label 100 comments. They agree on 85, so $P_o=0.85$ . Annotator A uses “toxic” 30% of the time and B 40%. Expected chance agreement is:

P_e=(0.30)(0.40)+(0.70)(0.60)=0.54

\kappa=\frac{0.85-0.54}{1-0.54}=\frac{0.31}{0.46}\approx0.674

The subtraction removes chance agreement; division by $1-P_e$ scales by the maximum improvement above chance that remained possible. A kappa of 1 means perfect agreement, 0 means no improvement over the marginal baseline, and negative values indicate worse-than-expected agreement.

Kappa is diagnostic, not a universal quality grade. Prevalence, annotator population, task difficulty, and category count affect it. Fleiss’ kappa extends the idea to more than two annotators.

Hard Labels Throw Away Disagreement

If five of seven annotators vote toxic, majority vote yields 1. A soft label retains:

q(y=\text{toxic}\mid x)=\frac{5}{7}\approx0.71

Soft labels preserve ambiguity, distinguish 5–2 from 7–0, and can train probability estimates that better reflect human uncertainty. They do not resolve systematic bias: seven annotators with the same blind spot can agree perfectly and still encode the wrong policy.

Use expert adjudication for high-risk disagreement, new policy categories, and cases where guidelines reveal a real conceptual gap.

Noise Has Structure

Random mistakes

Misclicks, fatigue, or inattention dilute the learning signal.

Systematic bias

Annotators repeatedly treat one dialect or community differently.

Ambiguity

The available context supports more than one defensible judgement.

Language mismatch

Annotator fluency is insufficient for slang, dialect, or code-switching.

Class imbalance

Rare threats receive too little practice and quality control.

Malicious annotation

Workers game speed incentives or deliberately corrupt results.

A Label Noise Model

Let $y$ be the latent policy-consistent label and $\tilde y$ the observed annotation. A transition matrix describes how the process changes one into the other:

T_{ij}=P(\tilde y=j\mid y=i)

T=\begin{bmatrix}0.96&0.04\\0.15&0.85\end{bmatrix}

Here non-toxic comments are incorrectly labelled toxic 4% of the time, while toxic comments are missed 15% of the time. The asymmetry matters: the observed positive class is not just a slightly blurry version of truth.

Real noise depends on language, context, annotator, and policy, so a single transition matrix is only a starting mental model.

Spend Human Attention Where It Matters

Active learning: label a seed set, train a provisional model, send uncertain or diverse examples to humans, retrain, and repeat. It reduces annotation cost, but its selected queue is not a representative evaluation set.

Weak supervision: combine noisy sources such as keyword rules, moderator actions, and user reports. The sources can create useful provisional labels at scale, but source dependencies and biases must be modelled or audited.

representative random sample → unbiased measurement
uncertain active-learning sample → decision-boundary improvement
rare-policy sample → coverage of costly failures
expert escalation queue → resolve ambiguous definitions

Labels Expire

Policies evolve after world events, new misinformation patterns, emerging slurs, legal changes, and community feedback. A label must therefore include a policy version and annotation time. Old examples may need reinterpretation, relabelling, or retirement.

Monitor label prevalence, disagreement, appeals, overturn rates, and slice-level confusion over time. An apparent model drift can actually be label drift: humans are applying a new concept while the target schema still carries the old one.

Annotation, Agreement, And Soft Labels

from collections import Counter
from dataclasses import dataclass

@dataclass(frozen=True)
class Annotation:
    comment_id: str
    annotator_id: str
    policy_version: str
    label: str
    confidence: float

def soft_label(labels: list[str], positive="toxic") -> float:
    return sum(label == positive for label in labels) / len(labels)

def majority_vote(labels: list[str]) -> str:
    return Counter(labels).most_common(1)[0][0]

def cohens_kappa(a: list[str], b: list[str]) -> float:
    assert len(a) == len(b)
    labels = set(a) | set(b)
    po = sum(x == y for x, y in zip(a, b)) / len(a)
    pe = sum((a.count(k)/len(a)) * (b.count(k)/len(b)) for k in labels)
    return (po - pe) / (1 - pe) if pe < 1 else 1.0

def active_learning_queue(rows, low=0.4, high=0.6):
    return rows[(rows.p_toxic >= low) & (rows.p_toxic <= high)]

Interview Discussion

Does high agreement prove labels are correct?
No. Annotators can consistently apply a biased guideline or share the same blind spot.

When are soft labels preferable?
When disagreement reflects genuine ambiguity and calibrated uncertainty is useful downstream.

What should happen when kappa is low?
Slice disagreements, inspect examples, revise the schema or guidelines, retrain annotators, and pilot again—not merely average the noise.

Final Design Exercise

Write an operational definition of targeted insult.
Add one positive, one negative, and one borderline example.
Choose binary, multilabel, or hierarchical outputs and defend the trade-off.
Decide when an annotator may abstain and when expert review is required.
For votes 5–2, compute the hard and soft labels.
Explain why an active-learning queue cannot be the final test set.

Workshop deliverable

A versioned policy sheet, label schema, annotation guide, disagreement procedure, soft-label rule, quality dashboard, and escalation path for the same YouTube toxicity system.

The Three-Lesson Specification Is Complete

A model cannot learn a better concept than the one encoded in its labels. High-quality labels are often worth more than a larger model.

We still have not trained a model—and that is intentional. We have defined the decision, built the data supply chain, and specified the target. Only now is it sensible to ask how features can be computed consistently for training and online inference.