Representation Learning

Throughout the Representation section, the same idea kept returning. Models became more powerful when we changed how the data was represented.

Decision trees represented data as a hierarchy of decisions. Random Forests and Gradient Boosting combined many tree representations. SVMs searched for a robust linear boundary. Kernel methods manually transformed data into richer feature spaces.

That raises a deeper question. If better representations lead to better models, why are humans designing the representations at all?

Can a machine learn its own representation?

This is not just another algorithm. It is the conceptual bridge from classical machine learning to modern deep learning.

Imagine predicting whether Jessie orders a burger. The dataset contains hours since last meal, burger price, friends ordering burgers, and restaurant distance. These are raw features.

Jessie is not really responding to the number 8. She is responding to hunger. She is not responding directly to the number 4. She is responding to social influence. The problem is that nobody measured those concepts.

Representation learning is the process of automatically discovering useful features from data. The model learns internal descriptions that make the prediction task easier.

A child learning to recognize dogs does something similar. At first, the child sees colors and pixels. Over time, useful concepts such as ears, tails, noses, and fur emerge because they help distinguish dogs from other things.

The General Idea

Suppose the original input is $x$. Instead of predicting directly from $x$, we first learn a transformation:

The vector $z$ is the learned representation. The prediction is then made from $z$ rather than the raw input:

Combining the two functions gives:

The function $f$ learns useful features. The function $g$ uses those features to make predictions.

Worked Example

Suppose Jessie has raw input $x=(8,16,4)$, representing hours since eating, burger price, and friends ordering burgers. A learned representation might be:

These numbers no longer correspond to obvious physical quantities. They are coordinates in a learned feature space. Two people with similar burger-buying behavior should produce similar representations, even if their raw inputs differ.

Why It Works

The optimization objective has not changed. We still minimize prediction loss. The difference is that we now optimize both the representation and the predictor at the same time.

Previously, feature engineering fixed $f$ by hand and training learned only $g$. Representation learning lets training adjust both $f$ and $g$.

As training reduces the loss, the learned representation gradually becomes more useful for the task.

What Makes A Good Representation?

A good representation should:

Notice that human interpretability is not required. Some of the best representations are difficult to name, but they are still useful.

Why It Changed Machine Learning

For decades, progress often depended on better feature engineering. Researchers designed image descriptors, speech features, text features, and domain-specific transformations by hand.

Modern deep learning shifted the focus. Instead of hand-designing representations, we build models that learn representations automatically. This is one reason deep learning became so effective for images, speech, and natural language.

This sketch shows the shape of a one-hidden-layer network. Do not worry about backpropagation yet. The important idea is that the hidden layer is a learned representation.

import numpy as np

def relu(x):
    return np.maximum(0, x)

def sigmoid(x):
    return 1 / (1 + np.exp(-np.clip(x, -500, 500)))

class OneHiddenLayerNetwork:
    def __init__(self, input_dim, hidden_dim):
        self.W1 = 0.01 * np.random.randn(input_dim, hidden_dim)
        self.b1 = np.zeros(hidden_dim)
        self.W2 = 0.01 * np.random.randn(hidden_dim, 1)
        self.b2 = 0.0

    def forward(self, X):
        z = relu(X @ self.W1 + self.b1)      # learned representation
        y_hat = sigmoid(z @ self.W2 + self.b2)
        return z, y_hat

model = OneHiddenLayerNetwork(input_dim=3, hidden_dim=4)
z, predictions = model.forward(X)

print("hidden representation", z[:5])
print("predictions", predictions[:5])

During training, the hidden representations change. Examples that need similar predictions can move closer together in hidden space, even if their raw inputs look different.