Embeddings

Representation Learning taught us that a model can learn useful internal descriptions of data. Embeddings make that idea concrete.

An embedding is a learned vector that represents an object. The object might be a restaurant, user, movie, product, word, sentence, image, city, protein, or almost anything else we want a model to reason about.

If a model learns a useful representation, what does that representation usually look like? A vector.

Imagine building a restaurant recommendation system. If a user likes McDonalds, Burger King should probably be a better recommendation than Nobu. A human sees that immediately. To a computer, these are just different labels.

An embedding replaces each label with a learned vector. The individual dimensions do not need predefined meanings. What matters is the geometry: similar restaurants should land near each other.

McDonalds and Burger King have similar vectors. Nobu is far away. Olive Garden is somewhere between fast casual and sit-down dining. That vector is the embedding.

What Is An Embedding?

An embedding is a dense vector representation of an object. If the object is $x$, an embedding function maps it into a vector space:

The dimension $d$ might be 2, 16, 128, 768, or 4096. The important point is not the exact dimension. The important point is that nearby vectors represent similar objects.

Why Not One-Hot Encoding?

A one-hot vector gives every category its own coordinate. With four restaurants, the representation might look like this:

The problem is that all one-hot categories are equally far apart. The model cannot tell from the representation that McDonalds is more similar to Burger King than it is to Nobu.

Embeddings solve this by learning coordinates where distance means something.

The Embedding Matrix

Suppose there are $V$ possible restaurants and we choose embedding dimension $d$. We learn an embedding matrix:

Each row is one object embedding. For four restaurants and three dimensions:

If Burger King has ID 2, the embedding lookup simply returns row 2:

This lookup is efficient. A 768-dimensional embedding can be much cheaper than a one-hot vector with millions of dimensions.

Learning The Embeddings

Embeddings are not typed in by humans. They usually start as small random numbers. During training, gradient descent updates the rows that appear in the mini-batch.

If users who like McDonalds often click Burger King, the loss encourages those vectors to become useful in similar ways. Over many updates, restaurants that appear in similar contexts move closer together.

The model is never explicitly told that McDonalds is similar to Burger King. It discovers that because the similarity helps reduce prediction error.

Why It Works

Embeddings learn geometry. Instead of memorizing isolated labels, the model learns where objects should live in a vector space. Similar behavior becomes nearby geometry. Different behavior becomes distance.

This geometry allows generalization. If a new burger restaurant appears, a few updates can move its vector toward other burger restaurants. The model can then reuse what it already learned about nearby objects.

A Concrete Language Example

Word embeddings are learned the same way. A model sees words in context and adjusts vectors so words used in similar contexts become close.

The model was not handed a dictionary of grammar. It learned vectors that helped predict words from surrounding words. Semantic relationships emerged because they were useful for the training task.

Embeddings Versus Feature Engineering

Complexity

If there are $V$ objects and embedding dimension $d$, the embedding table contains:

parameters. Training usually updates only the rows for objects in the current mini-batch, which is why embeddings remain practical even with large vocabularies.

This PyTorch sketch learns restaurant and user embeddings for a recommendation task. A high score means the model predicts the user will like the restaurant.

import torch
import torch.nn as nn

class RestaurantRecommender(nn.Module):
    def __init__(self, num_users, num_restaurants, dim=32):
        super().__init__()
        self.user_embedding = nn.Embedding(num_users, dim)
        self.restaurant_embedding = nn.Embedding(num_restaurants, dim)

    def forward(self, user_ids, restaurant_ids):
        user_vec = self.user_embedding(user_ids)
        restaurant_vec = self.restaurant_embedding(restaurant_ids)
        score = (user_vec * restaurant_vec).sum(dim=1)
        return score

model = RestaurantRecommender(
    num_users=10_000,
    num_restaurants=50_000,
    dim=32,
)

user_ids = torch.tensor([17, 17, 42])
restaurant_ids = torch.tensor([3, 91, 3])
scores = model(user_ids, restaurant_ids)

print(scores)
print(model.restaurant_embedding.weight[3])

After training, you can visualize restaurant embeddings by reducing them to two dimensions with PCA or t-SNE. Similar restaurants should often cluster together if the training signal was useful.