Backpropagation

A neural network can now perform a forward pass. Given Jessie's restaurant visit, it might predict a 95 percent chance that Jessie orders a burger.

But suppose Jessie does not order one. The network made a mistake. Now the hard question appears.

A network may contain millions of weights. Which weights caused the mistake, and by how much should each one change?

Backpropagation answers that question. It computes the gradient of the loss with respect to every parameter in the network.

Why Guessing Is Hopeless

Imagine a small network with hunger, price, and social influence feeding a hidden neuron, which feeds an output neuron. If the prediction is wrong, should we change the hunger weight, the price weight, the output weight, or all of them?

In a modern network, doing this by trial and error is impossible. If there are 100 million parameters, changing one parameter at a time would require 100 million separate forward passes for one update.

We need one systematic pass that assigns responsibility to every parameter. That is the job of backpropagation.

Think of a group project. The final report is poor. To improve next time, you need to know what caused the problem. Did one person write incorrect facts? Did another make misleading graphs? Did someone else compute the statistics incorrectly?

The final mistake came from many intermediate contributions. Backpropagation moves backward through those contributions and assigns each one a share of responsibility.

In neural-network language, that share of responsibility is a gradient:

It means: if this weight increased by a tiny amount, how would the loss change?

Separate The Three Ideas

Students often blend the training process into one vague step. Keep these pieces separate:

Backpropagation does not update weights. It computes the gradients that an optimizer will later use to update weights.

Forward Pass First

Suppose the model predicts $\hat y=0.92$ and the true label is $y=1$. For binary classification, binary cross-entropy is:

Because $y=1$, this simplifies to:

The forward pass has computed the prediction and the loss. It has not explained how any weight should change.

The Gradient We Need

For a weight $w$, the quantity we need is:

If this derivative is positive, increasing the weight increases the loss, so gradient descent decreases the weight. If it is negative, increasing the weight decreases the loss, so gradient descent increases the weight.

The Chain Of Computations

For one neuron, the computation might be:

The loss depends on $a$. The activation depends on $z$. The weighted sum depends on $w$.

The weight affects the loss indirectly. The chain rule tells us how to multiply the indirect effects:

What Each Term Means

Each local derivative is simple. Backpropagation combines many simple local derivatives into the gradient for every parameter.

Worked Example

Use an intentionally simple computation so the arithmetic is visible:

Let $w=2$ and $x=3$. Then:

Now compute the derivative with respect to the weight:

This gradient says that increasing $w$ would increase the loss strongly. Gradient descent therefore moves $w$ downward.

Why It Is Efficient

Backpropagation stores intermediate values from the forward pass, then walks backward through the computational graph. Every operation contributes its local derivative once.

Instead of running one forward pass per weight, backpropagation computes gradients for all weights in roughly the same order of work as a forward pass. This efficiency is what makes training large neural networks feasible.

Matrix Version

In practice, layers are vectorized. A layer computes:

Backpropagation computes $\frac{\partial L}{\partial W}$ for the whole weight matrix at once. Modern frameworks automate this matrix calculus, but the underlying idea is still the chain rule.

This NumPy example trains a tiny network with one sigmoid neuron. It separates the forward pass, backward pass, and parameter update so the roles stay clear.

import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

def sigmoid_grad(a):
    return a * (1 - a)

x = np.array([[3.0]])
y = np.array([[1.0]])

w = np.array([[2.0]])
b = np.array([[-4.0]])
lr = 0.5

# Forward pass
z = w @ x + b
a = sigmoid(z)
loss = 0.5 * (a - y) ** 2

# Backward pass
dL_da = a - y
da_dz = sigmoid_grad(a)
dL_dz = dL_da * da_dz

dL_dw = dL_dz @ x.T
dL_db = dL_dz

# Gradient descent update
w = w - lr * dL_dw
b = b - lr * dL_db

print("prediction:", a.item())
print("loss:", loss.item())
print("dL/dw:", dL_dw.item())
print("updated w:", w.item())

Automatic Differentiation

Modern ML engineers rarely write these derivatives by hand. PyTorch builds a computational graph during the forward pass. Calling $\text{loss.backward()}$ traverses that graph backward and stores gradients on each parameter.

import torch

x = torch.tensor([[3.0]])
y = torch.tensor([[1.0]])

w = torch.tensor([[2.0]], requires_grad=True)
b = torch.tensor([[-4.0]], requires_grad=True)

z = x @ w.T + b
a = torch.sigmoid(z)
loss = 0.5 * (a - y).pow(2).mean()

loss.backward()

print(w.grad)
print(b.grad)

Frameworks automate backpropagation. They do not avoid it. Understanding the mechanism explains why gradients can vanish, explode, or fail to flow through certain operations.