Machine Learning from Scratch

The Problem

Logistic regression solved binary classification. It predicts the probability of one class, and the other class is whatever probability remains.

P(y=1)=p

P(y=0)=1-p

That works because there are only two possibilities. If an email has probability $0.8$ of being spam, then it has probability $0.2$ of not being spam.

Now imagine an image classifier with three possible labels.

Cat

Dog

Horse

Knowing the probability of Cat is not enough. The remaining probability must be shared between Dog and Horse. We need a model that produces one probability for every class, and those probabilities must add up to one.

From Scores To Probabilities

A multiclass linear model produces one raw score for each class. For Cat, Dog, and Horse, the model might produce:

z=[2.0,1.0,0.0]

These scores are not probabilities. They can be negative, they do not need to sum to one, and a score of $2$ does not mean a probability of $2$ . The scores only say which classes the model currently prefers.

Softmax converts those scores into probabilities.

p_k=\frac{e^{z_k}}{\sum_{j=1}^{K}e^{z_j}}

The score $z_k$ belongs to class $k$ . The term $e^{z_k}$ makes the score positive. The denominator adds the positive values for all classes, which normalises the result so the probabilities sum to one. The output $p_k$ is the predicted probability for class $k$ .

For $z=[2,1,0]$ , the positive scores are:

e^2\approx7.39,\qquad e^1\approx2.72,\qquad e^0=1

\sum_{j=1}^{3}e^{z_j}=7.39+2.72+1=11.11

p\approx\left[\frac{7.39}{11.11},\frac{2.72}{11.11},\frac{1}{11.11}\right]\approx[0.67,0.24,0.09]

The model now says Cat is most likely, Dog is possible, and Horse is unlikely.

Interactive Demo

Softmax explorer

Catscore 2.00 · probability 66.5%

Dogscore 1.00 · probability 24.5%

Horsescore 0.00 · probability 9.0%

True class

One hot target[0, 1, 0]

Correct probability0.245

Cross entropy1.408

One Hot Labels And Loss

To train the model, the true label also needs a vector form. If the true class is Dog, we write:

y=[0,1,0]

This vector is not a probability estimate. Each value $y_k$ is an indicator. It tells the loss function which class is correct.

Categorical cross entropy punishes the model for assigning low probability to the correct class.

L=-\sum_{k=1}^{K}y_k\log(p_k)

Using the Dog example with predicted probabilities $[0.67,0.24,0.09]$ , the loss becomes:

L=-(0\log0.67+1\log0.24+0\log0.09)

L=-\log0.24

Only the correct class contributes because every incorrect class is multiplied by zero. The loss is large when the model gives the true class a small probability.

Binary cross entropy

L=-\left[y\log(p)+(1-y)\log(1-p)\right]

Categorical cross entropy

L=-\sum_{k=1}^{K}y_k\log(p_k)

Both losses have the same spirit. They punish the model for putting too little probability on the true class.

Implementation

import numpy as np

def one_hot(y, num_classes):
    out = np.zeros((len(y), num_classes))
    out[np.arange(len(y)), y] = 1
    return out

def softmax(scores):
    shifted = scores - np.max(scores, axis=1, keepdims=True)
    exp_scores = np.exp(shifted)
    return exp_scores / np.sum(exp_scores, axis=1, keepdims=True)

def cross_entropy(probs, y_one_hot):
    n = len(probs)
    return -np.sum(y_one_hot * np.log(probs + 1e-12)) / n

class SoftmaxRegression:
    def __init__(self, lr=0.1, epochs=1000):
        self.lr = lr
        self.epochs = epochs

    def fit(self, X, y):
        n, d = X.shape
        k = int(np.max(y)) + 1
        self.W = np.zeros((d, k))
        self.b = np.zeros(k)
        Y = one_hot(y, k)

        for _ in range(self.epochs):
            scores = X @ self.W + self.b
            probs = softmax(scores)
            error = probs - Y

            self.W -= self.lr * (X.T @ error) / n
            self.b -= self.lr * np.mean(error, axis=0)

    def predict_proba(self, X):
        return softmax(X @ self.W + self.b)

    def predict(self, X):
        return np.argmax(self.predict_proba(X), axis=1)

    def accuracy(self, X, y):
        return np.mean(self.predict(X) == y)

Interview Discussion

Why does sigmoid work for binary classification?

One probability determines the other because the two probabilities must sum to one.

Why is softmax used for multiclass classification?

It converts many raw scores into positive probabilities that sum to one.

What is one hot encoding?

It represents the correct class with one 1 and all other classes with 0.

Why does categorical cross entropy only care about the true class?

The one hot vector multiplies all incorrect class log probabilities by zero.

Active Recall

1. Explain where softmax fits in the prediction pipeline.

2. Given scores [2, 1, 0], calculate the approximate softmax probabilities.

3. Given y = [0, 1, 0], simplify the categorical cross entropy loss.

4. Why are raw class scores not probabilities?