Logistic Regression

Linear regression predicts continuous numbers. That works when the target is a price, a temperature, or a number of sales. Many prediction problems ask for a yes or no answer instead.

• Will this email be spam?
• Will this customer cancel?
• Is this transaction fraudulent?

This is binary classification. We usually encode the two classes as $0$ and $1$. It is tempting to fit a line and treat the output as the probability of class $1$.

The problem is that a line can output any real number. A prediction of $-1.3$ cannot mean negative probability. A prediction of $2.4$ cannot mean a probability greater than one. Probabilities must always stay between zero and one.

We need a model that keeps the useful part of linear regression while producing valid probabilities. Logistic regression begins by changing what the line is allowed to model.

Logistic regression does not model the probability directly. It models the log-odds. To see why, start with odds.

If an event has probability $p=0.8$, then the probability it happens is $0.8$ and the probability it does not happen is $0.2$. The odds are $0.8/0.2=4$. The event is four times as likely to happen as not happen.

Odds are positive, but a linear model can be negative. Taking the logarithm gives log-odds, which can be any real number.

This is the modelling assumption. Linear regression assumes the target itself is a linear function of the features. Logistic regression assumes the log-odds of the positive class are a linear function of the features.

This choice is attractive because probabilities are trapped between zero and one, while log-odds can move across the whole number line. A linear model naturally fits the log-odds scale.

Now we need to turn the log-odds back into a probability. Let $z=w^T x+b$.

This final expression is the sigmoid function. It is not chosen arbitrarily. It appears when we decide that a linear model should explain log-odds.

The model can now output valid probabilities. The next question is how to choose the weights and bias so the probabilities match the labels we observed.

A probability model should give high probability to what actually happened. This idea is maximum likelihood.

Suppose a model makes these predictions for three examples.

For the first two examples, the observed label is $1$. The model assigned probabilities $0.9$ and $0.8$ to the observed outcome. For the third example, the model predicted $p=0.3$ for class $1$, but the observed label was $0$. The probability assigned to what happened is therefore $1-0.3=0.7$.

If the examples are treated as independent, the probability of seeing the whole dataset under this model is the product of those probabilities.

For one observation, there is a compact way to select the right probability.

This works because the label is either $0$ or $1$. When $y=1$, the first factor remains and the second factor becomes one.

When $y=0$, the first factor becomes one and the second factor remains.

Across a dataset, likelihood multiplies the probability assigned to every observed label.

Products of many probabilities become tiny very quickly because every factor is at most one. Tiny products are awkward for computers and hard to reason about. Logs solve this by turning products into sums.

We want the log-likelihood to be as large as possible. Most optimisation tools are written as minimisers, which means they search for the smallest value of an objective. To use the same tools, we minimise the negative log-likelihood instead.

Averaging over the number of examples gives binary cross-entropy. So cross-entropy is not a random penalty. It is exactly the negative log-likelihood for binary labels.

Cross-Entropy

Mean squared error treats probability mistakes too gently when the model is confidently wrong. Cross-entropy behaves like a betting score. The more confidence the model places on the wrong outcome, the more it should pay.

If the true label is $1$, the loss for one prediction is $-\log(p)$.

A prediction of $0.9$ is confident and correct, so the loss is small. A prediction of $0.01$ is confidently wrong, so the loss is much larger. The model is being taught that confident mistakes are worse than uncertain mistakes.

This implementation trains logistic regression with gradient descent. Each update moves the weights in the direction that reduces binary cross-entropy.

import numpy as np

class LogisticRegression:
    def __init__(self, lr=0.1, epochs=1000):
        self.lr = lr
        self.epochs = epochs

    def sigmoid(self, z):
        z = np.clip(z, -500, 500)
        return 1 / (1 + np.exp(-z))

    def fit(self, X, y):
        n, d = X.shape
        self.w = np.zeros(d)
        self.b = 0.0

        for _ in range(self.epochs):
            z = X @ self.w + self.b
            p = self.sigmoid(z)
            error = p - y

            self.w -= self.lr * (X.T @ error) / n
            self.b -= self.lr * np.sum(error) / n

    def predict_proba(self, X):
        return self.sigmoid(X @ self.w + self.b)

    def predict(self, X, threshold=0.5):
        return (self.predict_proba(X) >= threshold).astype(int)

    def binary_cross_entropy(self, X, y):
        p = np.clip(self.predict_proba(X), 1e-15, 1 - 1e-15)
        return -np.mean(y * np.log(p) + (1 - y) * np.log(1 - p))