Activation Functions

In the previous lesson, a neural network was built by stacking layers. Each layer computed a weighted sum and then applied an activation function:

The natural question is not which activation function to memorize first. The deeper question is why activation functions are needed at all.

If we remove the activation functions, does stacking many layers still make the network more powerful?

Surprisingly, no. Without activation functions, a deep neural network is equivalent to a single linear model. Activation functions are what let deep networks actually become non-linear.

The Collapse Problem

Suppose a two-layer network has no activation function. The first layer computes:

The second layer computes:

Substitute the first equation into the second:

Expand:

Now define:

Then the whole network becomes:

That is one linear layer. More parameters were used, but nothing fundamentally more expressive was created.

Imagine a factory where every worker can only multiply the incoming number by 2. Worker 1 doubles the number, Worker 2 doubles it again, and Worker 3 doubles it again. The whole factory multiplies by 8.

You could replace all three workers with one worker who multiplies by 8. The factory looks deeper, but its behavior is still simple.

Now give one worker a rule: if the number is negative, replace it with zero. Suddenly the whole factory cannot be simplified into one multiplication. The rule introduced a bend.

Activation functions are those bends. The linear step mixes information. The activation step changes the shape of the representation.

Why Non-Linearity Matters

Think back to XOR. A straight line cannot separate the classes in the original input space. But if a hidden layer can bend, gate, or reshape the representation, points that were inseparable can become separable.

Linear transformations can rotate, stretch, shrink, and shift space. Non-linear activations allow a network to warp space. That is why hidden layers can build representations that linear models cannot.

What An Activation Function Does

Every neuron performs two conceptual steps:

This pattern repeats through deep learning: linear transformation, non-linear activation, linear transformation, non-linear activation.

Sigmoid

The sigmoid function maps any real number into the interval $(0,1)$:

Large negative values become close to 0. Large positive values become close to 1. This makes sigmoid useful when the output should be interpreted as a binary probability.

The problem is saturation. When $z$ is very positive or very negative, the curve becomes almost flat. Its derivative is:

Near the flat regions, this derivative is close to zero, so gradient updates become tiny. In deep networks, this contributes to the vanishing gradient problem.

Tanh

The hyperbolic tangent maps inputs into $(-1,1)$:

Unlike sigmoid, tanh is centered around zero. Negative inputs produce negative outputs, positive inputs produce positive outputs, and values near zero remain near zero.

This zero-centered behavior often helps optimization compared with sigmoid, but tanh still saturates for very large positive or negative inputs.

ReLU

The Rectified Linear Unit is extremely simple:

If $z<0$, ReLU outputs 0. If $z>0$, ReLU outputs $z$.

ReLU became popular because it has three practical advantages:

• It does not saturate for positive values, so positive gradients stay strong.
• It is computationally cheap because it only requires a comparison.
• It creates sparse activations because many neurons output exactly zero.

Dead ReLU

ReLU also has a failure mode. If a neuron always receives negative pre-activations, it always outputs zero. Its gradient is also zero, so the neuron may stop learning.

This is called a dead ReLU. The neuron is still present in the network, but it no longer contributes useful signal.

Leaky ReLU

Leaky ReLU keeps a small slope for negative values:

Negative inputs are still suppressed, but not completely. Because the negative side has a small derivative, the neuron can still receive gradient updates.

GELU

Transformers often use Gaussian Error Linear Units, usually called GELU:

Here $\Phi(z)$ is the cumulative distribution function of the standard normal distribution. You do not need to memorize that definition to understand the intuition.

GELU softly decides how much of the input to keep. Large positive values pass through almost unchanged. Large negative values are mostly suppressed. Values near zero are partially kept rather than being cut off sharply.

Worked Example

Apply several activation functions to the same vector:

Comparison

Activation functions are ordinary functions applied element by element to arrays. The implementation is short, but the modeling consequences are enormous.

import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

def tanh(z):
    return np.tanh(z)

def relu(z):
    return np.maximum(0, z)

def leaky_relu(z, alpha=0.01):
    return np.where(z > 0, z, alpha * z)

def gelu(z):
    return 0.5 * z * (
        1 + np.tanh(np.sqrt(2 / np.pi) * (z + 0.044715 * z**3))
    )

z = np.array([-3.0, 1.0, 5.0])

print("sigmoid:", np.round(sigmoid(z), 3))
print("tanh:", np.round(tanh(z), 3))
print("relu:", relu(z))
print("leaky_relu:", leaky_relu(z))
print("gelu:", np.round(gelu(z), 3))

Where The Activation Goes

In a neural network, the activation comes immediately after each hidden layer linear transformation:

def forward(x, W1, b1, W2, b2):
    z1 = W1 @ x + b1
    a1 = relu(z1)

    z2 = W2 @ a1 + b2
    y_hat = sigmoid(z2)

    return y_hat

Hidden layers often use ReLU, Leaky ReLU, GELU, or another non-linear activation. The output layer depends on the task: sigmoid for binary classification, softmax for multiclass classification, and often no activation for regression.