Neural Networks

The perceptron gave us the simplest artificial neuron: compute a weighted sum, then make a decision. That was powerful enough for linearly separable data, but it failed on problems like XOR.

If one neuron can only draw one simple boundary, what happens when neurons help other neurons?

This is the key idea of a neural network. A single neuron is limited. A layer of neurons can detect several simple patterns at once. Another layer can combine those patterns into more useful patterns. Repeating this process creates representation learning.

The Limitation We Are Escaping

A single perceptron sees the original inputs and draws one linear decision boundary. For XOR, the positive examples are diagonal from each other, so no single straight line can separate the classes.

The answer is not to search harder for the perfect single line. The answer is to change the representation. A hidden layer can create useful intermediate features such as OR and AND. Once those features exist, XOR becomes easy.

Imagine asking one friend whether Jessie will order a burger. That friend tries to combine hunger, price, and social pressure into one decision. This is like one perceptron.

Now imagine asking a team. One friend estimates hunger. Another estimates whether the burger feels affordable. Another watches what the group is ordering. A final friend combines those opinions into the actual prediction.

That is a neural network. Hidden neurons do not directly output the final answer. They compute intermediate signals that make the final answer easier.

A Concrete Burger Network

Suppose the raw input is:

The entries mean Jessie has not eaten for 8 hours, the burger costs 15 dollars, and 4 friends are ordering burgers. A first hidden layer might transform these raw numbers into:

A second hidden layer could combine those signals into a broader concept such as high burger intent. The output layer then turns that representation into the final prediction.

Layers Build Representations

This is why neural networks belong after representation learning. A network is not merely a bigger predictor. It is a system that repeatedly changes the representation until the prediction becomes simpler.

One Neuron

A neuron still starts with the same weighted sum from the perceptron:

Then it applies an activation function. For now, use the sigmoid:

The value $a$ is called the activation of the neuron. It becomes an input to the next layer.

One Layer

A layer contains several neurons. If the layer has three neurons, it computes three weighted sums:

Instead of writing each neuron separately, collect all weights into one matrix:

Then apply the activation function element by element:

This is the first major mental shift. A neural-network layer is a vectorized group of neurons computing many feature detectors at the same time.

Multiple Layers

A network stacks these transformations. Each layer receives the previous layer representation:

The first layer sees the raw input. The second layer sees the first hidden representation. The output layer sees the final hidden representation.

Worked Forward Pass

Suppose the input has two features:

Use a hidden layer with two neurons:

The hidden pre-activation is:

Apply sigmoid to each entry:

These two numbers are not the final prediction. They are the hidden representation passed to the next layer. This computation from input to output is called the forward pass.

XOR With a Hidden Layer

A useful way to solve XOR is to build two hidden features:

The first hidden neuron activates when at least one input is on. The second activates when both inputs are on. The output can then compute:

In plain English: predict 1 when OR is true but AND is false. The hidden layer made the right representation, so the output neuron only needs a simple boundary.

Why Depth Helps

A single hidden layer with enough neurons can approximate any continuous function on a bounded region. This is the universal approximation idea. It says neural networks are expressive enough in principle.

It does not say the network will learn the function easily, that one hidden layer is always efficient, or that more parameters automatically generalize better. In practice, depth often helps because later layers can reuse features learned by earlier layers.

Parameter Count

If a layer has $n$ inputs and $m$ neurons, it has $mn$ weights and $m$ biases.

This is why modern neural networks can contain millions or billions of parameters: every connection has a learnable weight.

The Missing Ingredient

If every layer were only linear, stacking layers would not help. A product of linear transformations is still a linear transformation:

Activation functions are what prevent the whole network from collapsing into one linear model. That is why the next lesson focuses on non-linearity.

This implementation builds a neural network with one hidden layer. It does not train the weights yet. The goal is to understand the architecture and the forward pass.

import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

class NeuralNetwork:
    def __init__(self, input_dim, hidden_dim, output_dim):
        rng = np.random.default_rng(seed=0)
        self.W1 = rng.normal(0, 0.1, size=(hidden_dim, input_dim))
        self.b1 = np.zeros((hidden_dim, 1))
        self.W2 = rng.normal(0, 0.1, size=(output_dim, hidden_dim))
        self.b2 = np.zeros((output_dim, 1))

    def forward(self, x):
        # x has shape (input_dim, batch_size)
        z1 = self.W1 @ x + self.b1
        a1 = sigmoid(z1)

        z2 = self.W2 @ a1 + self.b2
        y_hat = sigmoid(z2)

        return y_hat, {
            "z1": z1,
            "a1": a1,
            "z2": z2,
        }

X = np.array([
    [0, 0, 1, 1],
    [0, 1, 0, 1],
])

network = NeuralNetwork(input_dim=2, hidden_dim=3, output_dim=1)
predictions, cache = network.forward(X)

print(predictions)

A Hand-Built XOR Network

To see the architecture solve something concrete, we can set the weights by hand. Training will come later with backpropagation.

import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

X = np.array([
    [0, 0, 1, 1],
    [0, 1, 0, 1],
])

k = 12

# Hidden neuron 1 approximates OR.
# Hidden neuron 2 approximates AND.
W1 = np.array([
    [k, k],
    [k, k],
])
b1 = np.array([
    [-0.5 * k],
    [-1.5 * k],
])

H = sigmoid(W1 @ X + b1)

# Output keeps OR and subtracts AND.
W2 = np.array([[k, -k]])
b2 = np.array([[-0.5 * k]])

y_hat = sigmoid(W2 @ H + b2)

print(np.round(H, 2))
print(np.round(y_hat, 2))

The important part is not the exact numbers. The important part is the structure: hidden neurons create useful features, and the output neuron combines those features.