Machine Learning from Scratch

EDA Found Patterns. How Do We Turn Them Into A Model?

Our housing table contains square footage, bedrooms, distance, and price. EDA can show that larger houses tend to cost more. It cannot yet express one house compactly, combine several features into a prediction, quantify uncertainty, or improve a bad prediction rule.

Three mathematical languages solve those problems:

Linear algebra

Store and combine many features.

Probability & statistics

Describe variability, evidence, and uncertainty.

Differential calculus

Measure how predictions and loss change.

The goal is not mathematics in isolation. Every object should answer an ML question.

The Dependency Map

Linear algebra first: the next lesson writes a house as a feature vector and predicts with a dot product. Probability second: logistic regression turns a score into a probability and learns through likelihood. Calculus last: optimisation follows derivatives and gradients to reduce loss.

x, X, w·x → linear prediction
random variables, P(A|B), likelihood → probabilistic classification
derivatives, partials, ∇L → gradient descent and backpropagation

Prerequisite check

In the housing table, identify one row, three features, and the target.
Why must a model be evaluated on unseen examples?
What did EDA tell us that it could not prove causally?

Three Mathematical Lenses

One house: x = [1.2 thousand sqft, 3 bedrooms, 2.1 km]

sqft weight: 0.3bedroom weight: 20distance weight: -5

w · x = 0.3(1.2) + 20(3) + -5(2.1) = 49.86

I. Linear Algebra: Represent And Combine

Scalars, vectors, and shape

A scalar is one number: price $y=480$ thousand dollars. A vector is an ordered list: one house with 1.2 thousand sqft, 3 bedrooms, and 2.1 km distance:

x=\begin{bmatrix}1.2\\3\\2.1\end{bmatrix}\in\mathbb R^3

Order is part of the schema. If a model expects [sqft, bedrooms, distance], swapping columns changes meaning. A vector’s dimension $d$ is the number of features.

Matrices are datasets

Stack $n$ row vectors into a feature matrix:

X=\begin{bmatrix}1.2&3&2.1\\0.85&2&4.2\\1.9&4&8.0\end{bmatrix}\in\mathbb R^{3\times3}

Rows are examples; columns are features. In general $X\in\mathbb R^{n\times d}$ , target vector $y\in\mathbb R^n$ , and parameter vector $w\in\mathbb R^d$ .

Dot Products Are Weighted Evidence

The dot product multiplies matching entries and sums:

w^Tx=w\cdot x=\sum_{j=1}^{d}w_jx_j

Let $w=[200,25,-8]$ and bias $b=50$ . For our house:

\hat y=w^Tx+b=200(1.2)+25(3)-8(2.1)+50=348.2

Each weight says how strongly its feature contributes, in the feature’s units. The bias is a baseline. This is exactly the computation in linear regression.

Faded example: compute the prediction for $x=[0.85,2,4.2]$ using the same parameters.

Reveal solution

200(.85)+25(2)−8(4.2)+50 = 236.4 thousand.

Matrix–Vector Multiplication Predicts A Batch

\hat y=Xw+b\mathbf 1

Each row of $X$ takes its own dot product with the same $w$ . The output has shape $n\times1$ . Shape checking prevents many implementation errors:

(n\times d)(d\times1)=(n\times1)

Transpose $X^T$ swaps rows and columns. It appears when accumulating how every example contributes to each feature’s gradient.

Geometry: Length, Distance, And Similarity

The L2 norm measures vector length:

\|w\|_2=\sqrt{\sum_jw_j^2},\qquad \|w\|_2^2=\sum_jw_j^2

Squared norms later penalise large weights in L2 regularisation; norms define SVM margins and distances. The dot product also measures alignment. Cosine similarity removes magnitude:

\cos(\theta)=\frac{x\cdot z}{\|x\|\|z\|}

This will reappear when embeddings retrieve semantically similar text. Feature scaling matters: sqft measured in raw feet can dominate bedrooms merely because its numbers are larger.

II. Probability & Statistics: Reason Under Uncertainty

Population, sample, and random variable

A population is the broader process we care about; a sample is observed data. A random variable maps an uncertain outcome to a number. Let $Y=1$ if a listing sells within 30 days and 0 otherwise. A Bernoulli distribution with parameter $p$ states:

P(Y=y)=p^y(1-p)^{1-y},\qquad y\in\{0,1\}

The distribution is a model of uncertainty—not a list of observed rows.

Mean, variance, and standard deviation

\bar x=\frac1n\sum_{i=1}^n x_i

s^2=\frac{1}{n-1}\sum_{i=1}^n(x_i-\bar x)^2,\qquad s=\sqrt{s^2}

Mean locates a centre; variance measures squared spread; standard deviation returns to original units. For prices [300, 400, 500], mean is 400, squared deviations are [10,000, 0, 10,000], sample variance is 10,000 and standard deviation is 100.

Conditional Probability Is Directional

Among 100 listings, 30 are below median price (A), 40 sell fast (B), and 18 satisfy both:

P(A\mid B)=\frac{P(A\cap B)}{P(B)}=\frac{18}{40}=0.45

P(B\mid A)=\frac{18}{30}=0.60

These are not interchangeable. “Of fast-selling homes, how many were cheap?” differs from “Of cheap homes, how many sold fast?” This same reversal drives confusion between precision and recall.

The multiplication rule rearranges the definition:

P(A\cap B)=P(A\mid B)P(B)=P(B\mid A)P(A)

Bayes’ Rule Updates Beliefs

P(A\mid B)=\frac{P(B\mid A)P(A)}{P(B)}

Suppose 10% of listings have a serious defect. An inspection flags 80% of defective homes but also flags 20% of sound homes. For 1,000 homes: 100 defective → 80 flagged; 900 sound → 180 flagged. Among 260 flags, only 80 are truly defective:

P(defect\mid flag)=\frac{80}{260}\approx0.308

The base rate matters. Later, classification metrics use the same logic: even a strong detector can produce many false positives when positives are rare.

Expectation And Expected Cost

\mathbb E[X]=\sum_x xP(X=x)

If a pricing decision has 70% chance of $10k gain and 30% chance of $5k loss:

\mathbb E[G]=0.7(10)+0.3(-5)=5.5\text{ thousand}

Expectation is a probability-weighted long-run average, not a promised outcome. Threshold selection later minimises expected false-positive and false-negative cost.

Independence, IID, And Correlation

Events A and B are independent if $P(A\cap B)=P(A)P(B)$ . Random variables can be dependent even when correlation is zero; correlation captures only linear association.

\operatorname{Cov}(X,Y)=\mathbb E[(X-\mu_X)(Y-\mu_Y)]

\rho_{XY}=\frac{\operatorname{Cov}(X,Y)}{\sigma_X\sigma_Y}

IID means examples are independently and identically distributed. Many derivations assume it. Family-linked houses, time series, repeated users, and changing markets violate it; grouped splits and time-aware evaluation respond to those violations.

Likelihood As A Function Of Parameters

Probability fixes a parameter and asks about outcomes. Likelihood fixes observed outcomes and compares parameter values. For outcomes [1, 1, 0] under Bernoulli probability $p$ :

L(p)=p\cdot p\cdot(1-p)=p^2(1-p)

$p=0.5$ gives .125; $p=2/3$ gives $4/27\approx.148$ . Maximum likelihood chooses the parameter making observed data most plausible.

Products of many small probabilities underflow, so logarithms turn products into sums:

\log(ab)=\log a+\log b

KL divergence $D_{KL}(P\|Q)$ measures discrepancy between distributions and will return in distillation, variational methods, and preference optimisation.

III. Differential Calculus: Measure Change And Improve

Functions and slopes

A model is a function. If $f(x)=x^2$ , then 2 maps to 4. Average slope between $x$ and $x+h$ is:

\frac{f(x+h)-f(x)}{h}

For $f(x)=x^2$ , $x=2$ , and $h=0.1$ , slope is $4.1$ . As $h\to0$ , it approaches 4—the instantaneous derivative.

f'(x)=\frac{df}{dx}=\lim_{h\to0}\frac{f(x+h)-f(x)}{h}

The derivative is local sensitivity: a small input change $\Delta x$ produces approximately $f'(x)\Delta x$ output change.

Derivative Rules You Will Reuse

Rule	Form	ML use
Constant	d(c)/dx = 0	Bias-independent terms vanish.
Power	d(xⁿ)/dx = nxⁿ⁻¹	Squared error derivative.
Sum	d(f+g)=f′+g′	Loss sums over examples.
Product	d(fg)=f′g+fg′	Composite parameter expressions.
Chain	d f(g(x))/dx=f′(g(x))g′(x)	Backpropagation through layers.
Exponential	d eˣ/dx=eˣ	Sigmoid and softmax.
Logarithm	d log x/dx=1/x	Cross-entropy and likelihood.

Partial Derivatives And Gradients

A loss depends on many parameters. A partial derivative changes one while holding the others fixed:

L(w_1,w_2)=(w_1-3)^2+2(w_2+1)^2

\frac{\partial L}{\partial w_1}=2(w_1-3),\qquad \frac{\partial L}{\partial w_2}=4(w_2+1)

The gradient stacks partial derivatives into the vector introduced in linear algebra:

\nabla L(w)=\begin{bmatrix}\partial L/\partial w_1\\\partial L/\partial w_2\end{bmatrix}

At $w=(0,0)$ , gradient is $[-6,4]^T$ . It points in the steepest local increase; the negative gradient points downhill.

Optimisation Preview

w_{new}=w_{old}-\eta\nabla L(w_{old})

Learning rate $\eta$ controls step size. From $w=(0,0)$ with $\eta=0.1$ , the update is:

w_{new}=\begin{bmatrix}0\\0\end{bmatrix}-0.1\begin{bmatrix}-6\\4\end{bmatrix}=\begin{bmatrix}0.6\\-0.4\end{bmatrix}

This is not yet the full gradient-descent lesson. The essential bridge is: training defines a loss, calculus measures how loss changes, and optimisation moves parameters in a direction expected to shrink it.

The Three Languages In NumPy

import numpy as np

# Linear algebra: batch predictions
X = np.array([[1.2, 3, 2.1], [.85, 2, 4.2], [1.9, 4, 8.0]])
w = np.array([200., 25., -8.])
predictions = X @ w + 50

# Statistics: centre and spread
prices = np.array([300., 400., 500.])
mean = prices.mean()
sample_variance = prices.var(ddof=1)

# Probability: likelihood of observations [1, 1, 0]
p = 2/3
likelihood = p**2 * (1-p)
log_likelihood = 2*np.log(p) + np.log(1-p)

# Calculus: numerical derivative of f(x)=x²
def derivative(f, x, h=1e-5):
    return (f(x+h)-f(x-h))/(2*h)

print(derivative(lambda x: x**2, 2.0))  # approximately 4

Mastery Gate

1. Given X ∈ ℝ^(50×4) and w ∈ ℝ^4, state the shape of Xw.

2. Compute one three-feature dot product without notes.

3. Explain variance in words and compute it for [1, 2, 3].

4. Construct a 2×2 table where P(A|B) differs from P(B|A).

5. Compute a Bernoulli likelihood for [1,0,1,1].

6. Differentiate (w-4)^2 and take one gradient step.

7. Tomorrow retrieve dot product, conditional probability, expectation, derivative, and gradient. Three days: redo the worked problems. One week: explain how all three subsections feed training.

Common Mistakes

Shape blindness: Multiplication is invalid unless inner dimensions match.
Scale blindness: Large-unit features dominate dot products and distances.
Conditional reversal: P(A|B) is not P(B|A).
Base-rate neglect: Detector quality alone does not determine posterior probability.
Likelihood confusion: Likelihood is not a probability distribution over parameters unless a prior is added.
Correlation as causation: Association does not prove intervention effect.
Derivative as finite difference: A derivative is the limiting local slope, not just one secant.
Gradient as update: The gradient points uphill; gradient descent subtracts it.

Now We Can Build A Predictor

A house is a vector, a dataset is a matrix, a prediction can be a dot product, uncertainty can be quantified, and a loss can be reduced with gradients. Linear Regression now has all of its prerequisite language.