Is This Actually An ML Problem?
YouTube receives millions of comments every day. Users complain that toxic comments remain visible for too long. Management asks: “Build an AI that removes toxic comments.”
Pause: is that request precise enough to build?
No. “Toxic” has no operational definition. “Removes” skips the decision policy. The request names no stakeholders, costs, constraints, or success measure. The first job of an ML engineer is not writing code; it is translating a vague product desire into a precise engineering problem.
Most machine-learning failures begin with solving the wrong problem.
From A Healthier Community To A Decision
Product goal: make YouTube conversations healthier. ML goal: estimate whether a comment violates a toxicity policy. These goals are related, but not equivalent.
A classifier can become more accurate while the product becomes worse: it may improve on abundant easy comments but miss threats; remove more borderline comments and drive up appeals; or respond too slowly to reduce time-to-removal. Accuracy measures predictions. The product goal concerns people and outcomes.
Learning objectives
- Separate product and ML problems
- Identify stakeholders and constraints
- Specify observable inputs and actionable outputs
- Define success metrics and failure costs
- Design human review and abstention
- Explain why specification precedes collection
Product-Decision Workshop
Click the production pipeline
Toxic comments remain visible too long.
Decision threshold
Move the threshold and watch product costs move with it.
Precision
89%
Recall
54%
Auto-removed / day
1,840
Human reviews / day
2,550
A higher threshold protects legitimate speech but leaves more abuse online. A lower threshold removes abuse faster but raises appeals and wrongful removals.
Policy lab: what should the system do?
“Thanks for explaining that so clearly!”
“You're absolutely brilliant. Nice job ruining everything.”
“I disagree—this argument ignores the evidence.”
“You're an id10t and nobody wants you here.”
“The character says, ‘I hate you,’ in the final scene.”
Stakeholders Have Conflicting Objectives
| Stakeholder | Goal | Failure they feel |
|---|---|---|
| Viewer | Participate without harassment | Abuse remains visible |
| Creator | Healthy community and expressive freedom | Fans are wrongly silenced |
| Moderator | A manageable, well-prioritised queue | Volume and traumatic exposure |
| Advertiser | Brand-safe placement | Overly broad removals shrink reach |
| Platform | Trust, retention, and sustainable cost | Incidents, churn, and expensive review |
| Policy team | Consistent enforcement | Model behaviour diverges from written policy |
| Legal team | Comply across jurisdictions | Illegal content or unlawful removal |
There is no threshold that maximises all of these goals. Product engineering makes the conflict explicit and chooses a defensible operating point.
Define What The System Can Observe
Comment text
Direct wording, spelling, and phrasing—but not intent.
Reply chain
Conversation context, at higher latency and privacy cost.
Language
Routes specialised policy and models; language identification can fail.
Video category
Separates gaming banter from children’s content, but can encode stereotypes.
User history
May reveal repeated harassment, but raises privacy and fairness concerns.
Time
Supports incident response and policy versioning, not future outcomes.
Every input must exist at decision time. An appeal result or future moderator action cannot be an online feature. If context is unavailable, the system should express uncertainty rather than invent it.
Define An Output That Supports Action
A binary label is simple but hides uncertainty. Five toxicity categories support policy-specific action but require more annotation. A severity score captures degree but may be hard to define consistently. A probability supports thresholds, while recommend human review lets the system abstain.
0.35–0.80 → moderator queue
0.80–1.00 → automatic action, with appeal
The most useful model output is often not the final product decision. It is evidence consumed by a decision policy.
Failure Taxonomy
False positive
‘I disagree with every point’ is removed.
Legitimate speech is suppressed; creators appeal.
False negative
‘Nobody wants your kind here’ stays visible.
A viewer experiences abuse before review.
Context failure
A documentary quote is treated as the speaker’s own attack.
Quoted language is mistaken for intent.
Sarcasm
‘Brilliant—another video full of lies.’
Literal wording hides a hostile pragmatic meaning.
Reclaimed language
An in-group term is classified without speaker/community context.
The same token can have different social meaning.
Cross-language
A dialect or code-switched insult is missed.
Coverage differs across languages and communities.
Adversarial spelling
‘y0u ar3 an 1d10t.’
Users adapt their writing to evade filters.
Policy mismatch
The model removes profanity although policy allows non-targeted profanity.
The prediction encodes the wrong rule.
Prediction, Decision, And Objective
For observable context , the model estimates:
A simple automatic action uses threshold :
But a production system usually has two thresholds:
The thresholds should minimise expected product cost, not maximise accuracy:
The costs depend on policy, reversibility, language, severity, and stakeholder impact.
ML Metrics Are Proxies
A system can reach 99% accuracy by predicting “not toxic” when toxicity is rare. Connect the confusion matrix to product measures: precision influences wrongful-removal and appeal rates; recall influences how much abuse remains; latency influences time-to-removal.
Precision, recall, calibration, slice recall
User reports, creator satisfaction, appeal rate, moderator workload, time-to-removal
A proxy becomes dangerous when the team optimises it after it stops tracking the true goal.
Implement The Specification—Not A Model
from dataclasses import dataclass
from enum import Enum
class Action(Enum):
KEEP = "keep"
REVIEW = "review"
REMOVE = "remove"
@dataclass(frozen=True)
class ProblemSpec:
product_goal: str
policy_version: str
observable_inputs: tuple[str, ...]
low_threshold: float
high_threshold: float
primary_metrics: tuple[str, ...]
guardrails: tuple[str, ...]
SPEC = ProblemSpec(
product_goal="Reduce exposure to policy-violating comments",
policy_version="toxicity-v1.0",
observable_inputs=("text", "reply_chain", "language", "video_category"),
low_threshold=0.35,
high_threshold=0.80,
primary_metrics=("severe_toxicity_recall", "appeal_rate", "time_to_removal"),
guardrails=("language_slice_floor", "human_review_band", "appeal_path"),
)Policy, Not Morality
The model does not discover a universal definition of good speech. It predicts how a versioned platform policy would classify an example. Cultural norms differ, policies change, and legal requirements vary. That makes policy provenance, appeals, subgroup evaluation, and human oversight part of the system.
Design checkpoint: before collecting a single example, write the action, affected stakeholders, observable evidence, false-positive and false-negative costs, abstention rule, and success measures.
Interview Discussion
Why not optimise accuracy?
It weights every example equally and ignores action costs, imbalance, latency, and stakeholder impact.
When should the model abstain?
When context is missing, uncertainty is high, or the consequence is costly and difficult to reverse.
Why define the problem before collecting data?
Inputs, labels, sampling, privacy rules, and evaluation all depend on the exact decision being supported.
Active Recall And Practice
- State the product goal and ML task without treating them as equivalent.
- Name one conflict between creators and moderators.
- Why can an appeal outcome not be an online input?
- Give a context failure and the safest decision for it.
- Sketch a two-threshold human-review policy.
- Choose one model metric and one product metric that should move together.
What Comes Next?
Problem definition determines everything that follows; a larger model cannot repair the wrong objective. We now know the decision we want to support. Next: where do ten million representative, privacy-safe examples come from?