Machine Learning from Scratch

The Problem

Linear and logistic regression are simple models. That simplicity is useful, but it also limits what they can express. The next question is natural.

Can a simple model perform better if we give it better features?

A feature is a measurable input used by the model. For house prices, features might include square footage, number of bedrooms, distance to the city centre, and age of the house.

Feature engineering means transforming raw data into inputs that make the target easier to predict. The model may stay linear, but the representation becomes more useful.

Raw And Engineered Features

A linear model cannot naturally represent curvature unless the curved transformation is given to it. If the true relationship bends, a model using only $x$ may fail. Adding $x^2$ lets the same linear model fit a quadratic pattern.

x\rightarrow x^2

Interaction features let the model express that two inputs matter together.

x_1,x_2\rightarrow x_1x_2

Ratio features are useful when the relationship depends on proportion rather than absolute size.

\frac{\text{income}}{\text{debt}}

Text

token count, TF IDF, embeddings, language

Temporal

day of week, time since last event, seasonality

Categorical

one hot encoding, target encoding, learned embeddings

Domain

business rules, physical constraints, expert signals

Feature engineering is not a trick for avoiding better models. It is often the first serious modelling decision. Later, representation learning will ask whether a model can learn useful features automatically.

Interactive Demo

Feature set comparison

Raw features

square footageage of house

The model sees size and age, but it cannot represent curved effects or useful combinations.

Training accuracy74%

Validation accuracy69%

Data Leakage

Better features are only valid if they would exist when the model makes a real prediction. This is the central rule.

\text{A feature must be available at prediction time.}

Data leakage happens when training uses information from the future, the label, or the evaluation process. The model appears strong during development and then fails in production.

Task	Leaky feature	Why it leaks
Toxicity detection	whether a moderator later deleted the comment	the decision happens after prediction
Churn prediction	whether the customer eventually cancelled	it is the answer, not an input
Risk scoring	post outcome data	the model would not know it beforehand

In a YouTube toxicity model, useful features can include the comment text, language, parent comment embedding, account age, previous moderation history, and creator settings. Leaky features include reports received after posting, moderator decisions after posting, and appeal outcomes.

Good features should be predictive, available at inference time, stable, cheap to compute, ethically defensible, and not direct proxies for protected attributes unless there is an explicit and justified reason.

Implementation

import numpy as np

def add_features(X):
    x1 = X[:, 0]
    x2 = X[:, 1]
    engineered = np.column_stack([
        x1,
        x2,
        x1 ** 2,
        x1 * x2,
        x1 / (np.abs(x2) + 1e-6),
    ])
    return engineered

def sigmoid(z):
    return 1 / (1 + np.exp(-np.clip(z, -500, 500)))

def train_logistic(X, y, lr=0.1, epochs=1000):
    w = np.zeros(X.shape[1])
    b = 0.0
    for _ in range(epochs):
        p = sigmoid(X @ w + b)
        error = p - y
        w -= lr * (X.T @ error) / len(X)
        b -= lr * np.mean(error)
    return w, b

def accuracy(X, y, w, b):
    return np.mean((sigmoid(X @ w + b) >= 0.5) == y)

# Split first, engineer second, and never use future information.
X_train_raw, X_val_raw = X[:80], X[80:]
y_train, y_val = y[:80], y[80:]

w_raw, b_raw = train_logistic(X_train_raw, y_train)
X_train_eng = add_features(X_train_raw)
X_val_eng = add_features(X_val_raw)
w_eng, b_eng = train_logistic(X_train_eng, y_train)

print("raw validation", accuracy(X_val_raw, y_val, w_raw, b_raw))
print("engineered validation", accuracy(X_val_eng, y_val, w_eng, b_eng))
print("engineered coefficients", w_eng)

Interview Discussion

What features would you build for this product?

Start from the prediction moment, list what is known then, and choose signals that are predictive, stable, cheap, and defensible.

How do you detect data leakage?

Ask whether the feature would exist at inference time and whether it was caused by the label or a later decision.

Why can feature engineering outperform changing the model?

A simple model with the right representation may express the real pattern more easily than a complex model with poor inputs.

What features are available for a first toxicity model?

Comment text, language, creator settings, parent comment context, and historical moderation signals known before the new decision.

Active Recall

1. Give three examples of engineered features.

2. Explain why x squared can help a linear model.

3. Identify which features leak in a list of candidate inputs.

4. Why should validation accuracy be checked after adding engineered features?