The Titanic Dataset Looks Ready—It Is Not
We have 891 passenger rows, 12 columns, and a survival label. That sounds like a supervised classification dataset. But 177 ages and 687 cabins are missing, fare is highly skewed, ticket values repeat across families, and passenger class is entangled with fare and cabin.
EDA is the disciplined search for data problems, plausible patterns, and questions a model will inherit.
We are not trying to prove why anyone survived. We are learning what can responsibly be described and what needs further analysis.
Retrieval Warm-Up
- Classify Age, Pclass, Embarked, Name, and Survived by type and role.
- Why is survival prediction supervised?
- Why must transformations be learned from training data only?
A repeatable EDA sequence
Titanic Pattern Explorer
This is an association, not proof that changing the group would cause survival. Class, age, family structure, and evacuation policy are entangled.
Missing-age audit
177 of 891 ages are missing.
Age analysis status: incomplete—missingness must be reported
1. Establish Shape, Grain, And Quality
Grain: one row represents one passenger—not one family, ticket, or cabin. The target is Survived.
Rows
891
Columns
12
Survived
342 (38.4%)
Duplicates by row
0
PassengerId is unique, but repeated tickets reveal related travel groups. Missingness is not uniform: Cabin is absent for 77%, Age for 19.9%, Embarked for 0.2%. A missing cabin may reflect passenger class and record-keeping, not random deletion.
2. Inspect Each Distribution
| Column | Finding | Implication |
|---|---|---|
| Survived | 38.4% positive | Accuracy alone has a 61.6% majority baseline. |
| Pclass | 55% are third class | Aggregate results are dominated by third class. |
| Age | median 28; range 0.42–80; 177 missing | Report missingness and avoid treating missing as zero. |
| Fare | median 14.45; maximum 512.33 | Strong right skew and extreme values. |
| Cabin | 687 missing | Too sparse for naïve use; missingness itself is informative. |
| Embarked | S dominates; 2 missing | Small categories need counts beside rates. |
3. Surface A Pattern Without Overclaiming
Observed survival rates are about 74% for female passengers versus 19% for male passengers, and 63%, 47%, and 24% for first, second, and third class respectively. These are large associations.
They are not isolated causal effects. Sex, class, age, fare, cabin location, family group, and evacuation procedures interact. Compare counts and uncertainty, slice combinations, and write “associated with” rather than “caused.”
Simpson’s-paradox warning: an aggregate relationship can change or reverse after stratifying by another variable. Always inspect meaningful subgroups with adequate support.
4. Missingness, Outliers, And Leakage
Ask why it is missing, compare missing/non-missing groups, preserve an indicator, and learn imputations on train only.
A fare of 512 may be valid, not an error. Verify provenance before clipping.
Boat assignment or rescue record would reveal survival but would not exist at the intended prediction time.
Reproducible EDA In Pandas
import pandas as pd
df = pd.read_csv("notebook/data/titanic.csv")
assert len(df) == 891
profile = pd.DataFrame({
"dtype": df.dtypes.astype(str),
"missing_n": df.isna().sum(),
"missing_pct": df.isna().mean().mul(100).round(1),
"unique": df.nunique(dropna=False),
})
survival_by_sex = df.groupby("Sex")["Survived"].agg(["count", "mean"])
survival_by_class = df.groupby("Pclass")["Survived"].agg(["count", "mean"])
age_missing_slice = df.assign(age_missing=df.Age.isna()).groupby("age_missing")["Survived"].agg(["count","mean"])
print(profile)
print(survival_by_sex)
print(survival_by_class)EDA Report: Answer First
Finding
survival is strongly associated with sex and passenger class.
Evidence
female 74% vs male 19%; first class 63% vs third 24%, with group counts reported.
Caveats
observational associations, confounding, missing age/cabin, and historical context.
Next check
stratify sex by class and age; create train/validation/test splits before learning preprocessing.
Final Foundations Exercise
1. State the dataset grain, target, and learning type.
2. List the quality checks in order without notes.
3. Explain why 74% versus 19% is not a causal conclusion.
4. Choose a treatment for missing age and defend what information it preserves.
5. Name one plausible leakage column.
6. Tomorrow reproduce the EDA sequence; in one week write the four-part finding from memory.
Why We Now Need Mathematics
EDA revealed differences, skew, uncertainty, and interacting variables. To turn patterns into predictions, we need a precise language for weighted inputs, error, probability, and improvement. That is the bridge into the Prediction unit.