Convolutional Neural Networks

Fully connected neural networks work naturally for flat vectors: age, salary, price, hours since last meal. But images are not just lists of numbers. They are grids.

A small RGB image with shape $224\times224\times3$ has more than 150,000 input values. Flattening it destroys the fact that nearby pixels are related.

Why should a network looking at images pretend that every pixel is unrelated to every other pixel?

A convolutional neural network, or CNN, is designed around the spatial structure of images. It looks for local patterns, reuses the same pattern detectors everywhere, and gradually builds visual concepts from simple to complex.

What Flattening Throws Away

Suppose we want to detect whether Jessie is holding a burger. The image contains edges, colors, hands, bread, shadows, and background. When we flatten pixels into one long vector, the model no longer naturally knows which pixels were neighbors or which pixels formed an edge.

A fully connected network can still learn from images, but it must rediscover image geometry from scratch. CNNs build that geometry into the architecture.

Imagine reading a book. You do not understand the whole page all at once. Letters form words. Words form phrases. Phrases form sentences. Meaning becomes more abstract layer by layer.

CNNs do something similar for images. Early layers detect tiny visual patterns such as edges, corners, and curves. Later layers combine those into textures, object parts, and eventually whole objects.

The Three Core Ideas

Local Receptive Fields

A neuron does not need to inspect the entire image to detect a small edge. It only needs a local patch. This patch is called its receptive field.

Nearby pixels are much more related than distant pixels. A corner, edge, or texture is formed by local neighborhoods, not by unrelated pixels on opposite sides of an image.

Weight Sharing

If the network learns a detector for a horizontal edge, it should not need a separate detector for every possible location. A horizontal edge is still a horizontal edge in the top-left, center, or bottom-right.

CNNs reuse the same filter everywhere. That dramatically reduces parameters and lets the model recognize a pattern wherever it appears.

Convolution

A convolution slides a small filter across an input image. At each location, it multiplies the local patch by the filter element by element and sums the result.

Here $X$ is the input image, $K$ is the filter, and $Y$ is the output feature map.

Worked Example

Suppose the image patch is:

and the filter is:

The output at that location is:

The filter then slides to the next location and repeats this calculation. The collection of all responses is the feature map.

Feature Maps

Each filter produces one feature map. If a layer learns 32 filters, it produces 32 feature maps. One might respond to vertical edges, another to corners, another to a texture pattern.

These feature maps become the input to the next convolutional layer, where simple patterns are combined into more complex patterns.

Stride

Stride controls how far the filter moves after each computation. Stride 1 examines neighboring positions. Stride 2 skips every other position and produces a smaller feature map.

Larger strides reduce computation and spatial resolution.

Padding

Without padding, images shrink after convolution. A $5\times5$ image with a $3\times3$ filter produces a $3\times3$ output.

Padding adds zeros around the border so the convolution can preserve spatial size. This is especially useful in deep CNNs, where repeated shrinking would make feature maps tiny too quickly.

Pooling

Max pooling keeps the strongest activation in a local region:

Average pooling would return $2.5$. Pooling reduces spatial resolution and makes the representation less sensitive to tiny shifts.

Parameter Count

A fully connected layer from a $224\times224$ image to 1000 hidden units needs:

A single $3\times3$ convolutional filter has only 9 weights, reused across the entire image. With RGB inputs, it has $3\cdot3\cdot3=27$ weights. That reuse is the power of weight sharing.

Translation Equivariance And Invariance

Convolutions are translation equivariant: if the image shifts, the feature map shifts in the same way. The filter still detects the same pattern in the new location.

Pooling and later classification layers provide approximate translation invariance: the final prediction becomes less sensitive to small shifts in the object's exact position.

A minimal image classifier combines convolution, activation, pooling, and a final fully connected classifier.

import torch
from torch import nn

class SmallCNN(nn.Module):
    def __init__(self):
        super().__init__()
        self.features = nn.Sequential(
            nn.Conv2d(in_channels=1, out_channels=16, kernel_size=3, padding=1),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2),
            nn.Conv2d(in_channels=16, out_channels=32, kernel_size=3, padding=1),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2),
        )
        self.classifier = nn.Sequential(
            nn.Flatten(),
            nn.Linear(32 * 7 * 7, 10),
        )

    def forward(self, x):
        x = self.features(x)
        return self.classifier(x)

model = SmallCNN()
images = torch.randn(32, 1, 28, 28)
logits = model(images)

print(logits.shape)  # (32, 10)

For MNIST, the input shape is batch size by channel by height by width. The convolutional layers produce feature maps; the classifier turns those feature maps into class scores.

Visualizing Learned Filters

After training, the first convolutional layer contains filters that often resemble edge or stroke detectors. Visualizing those filters and their feature maps is one of the best ways to see representation learning in a CNN.