Sequence Models

So far, our neural networks have assumed that the input is a fixed collection of numbers. Tabular features are vectors. Images are grids. But many important problems arrive one step at a time.

Consider two messages:

The only difference is the word not, but the meaning changes completely. A model cannot understand the sentence by treating words as independent pieces.

How can a neural network remember what it has already seen?

Why Fixed-Vector Networks Struggle

Suppose we flatten a sentence into one vector:

This creates three problems.

Sequence models are designed for this setting: ordered inputs where the current interpretation depends on what came before.

Imagine reading a novel. You do not wait until you have read all 500 pages before understanding the first page. You update your understanding as each sentence arrives.

A sequence model works the same way. At every step, it receives the current input and its previous memory. Then it produces an updated memory and, optionally, an output.

What Is A Sequence?

A sequence is an ordered collection. The order is the point.

The Core Computation

The model repeatedly applies the same update:

This repeated computation is the defining idea behind recurrent sequence models.

Recurrent Neural Networks

The simplest neural sequence model is the recurrent neural network, or RNN. At time step $t$, it receives the current input $x_t$ and the previous hidden state $h_{t-1}$.

The hidden state $h_t$is the model's memory: a running summary of what has been seen so far.

What Each Term Means

Worked Example

Suppose we simplify everything to one-dimensional values. Initially, $h_0=0$. Let $W_x=2$ and $W_h=0.5$.

For the first word, represent Jessie as $x_1=1$:

For the second word, represent likes as $x_2=0.8$:

The second memory depends on both the current word and the memory from the previous word. That recursive dependency is what makes the model sequential.

Unrolling Through Time

RNNs are often drawn as one box, but during computation we unroll that box across time:

It looks like many networks, but every time step uses the same weights. This is called parameter sharing through time. The same grammar-like computation is reused at every position in the sequence.

Sequence Task Types

The Weakness Of Vanilla RNNs

The hidden state gave neural networks memory, but it is an imperfect memory. If an important word appears many time steps ago, its influence must survive repeated transformations.

In long sentences, ordinary RNNs gradually forget. This is the long-term dependency problem, and it motivates LSTMs and GRUs in the next chapter.

This minimal PyTorch model processes a sequence one step at a time. The RNN produces hidden states, and the final hidden state is used for a many-to-one classification task such as sentiment analysis.

import torch
from torch import nn

class SimpleRNNClassifier(nn.Module):
    def __init__(self, vocab_size, embed_dim, hidden_dim, num_classes):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, embed_dim)
        self.rnn = nn.RNN(
            input_size=embed_dim,
            hidden_size=hidden_dim,
            batch_first=True,
        )
        self.classifier = nn.Linear(hidden_dim, num_classes)

    def forward(self, token_ids):
        # token_ids shape: (batch_size, sequence_length)
        x = self.embedding(token_ids)

        # outputs contains every hidden state.
        # final_hidden contains the last hidden state.
        outputs, final_hidden = self.rnn(x)

        last_memory = final_hidden[-1]
        logits = self.classifier(last_memory)
        return logits

model = SimpleRNNClassifier(
    vocab_size=10000,
    embed_dim=64,
    hidden_dim=128,
    num_classes=2,
)

batch = torch.randint(0, 10000, (32, 20))
logits = model(batch)
print(logits.shape)  # (32, 2)

For next-token prediction, the classifier can be applied to every hidden state instead of only the final one. The same RNN machinery supports many sequence task shapes.