Tokenisation

A transformer never sees words directly. It performs matrix multiplication on numbers. So before it can process a sentence, language must be translated into a sequence of integer IDs.

How can a transformer process human language when computers only understand numbers?

Tokenisation is that translation process. It decides how text is split into units and assigns each unit an integer ID from a fixed vocabulary.

Why This Is Necessary

Humans see I want a burger. and immediately notice words. A computer initially stores characters as bytes, such as hexadecimal values or binary digits. Nothing in those raw bytes tells the model that burger is a meaningful unit.

Before attention, embeddings, or gradient descent can happen, text needs a numerical representation.

Tokenisation is a compromise. If tokens are too small, sequences become long and expensive. If tokens are too large, the vocabulary explodes and rare words become impossible to handle.

Attempt 1: Characters

Character-level tokenisation gives every character an ID:

This always works, but it makes sequences much longer. A word like internationalisation becomes more than twenty tokens. Since attention compares tokens with tokens, longer sequences quickly become expensive.

Attempt 2: Words

Word-level tokenisation is shorter:

But word vocabularies explode. Names, slang, code, URLs, scientific terms, new words, and many languages create too many possibilities. A pure word-level tokenizer does not gracefully handle a word it has never seen.

Attempt 3: Subwords

Modern LLMs usually use subword tokenisation. Frequent words can stay whole, while rare words split into reusable pieces:

Subwords balance coverage and efficiency. They can represent new text without needing a vocabulary entry for every possible word.

What Is A Token?

A token is one unit chosen by a tokenizer. It might be a word, a word piece, a punctuation mark, whitespace, a number, or an emoji.

Modern tokenizers often include leading spaces inside tokens. For example, New can be a different token from New. This surprises beginners, but it helps the tokenizer represent common text patterns efficiently.

Vocabulary

A tokenizer has a fixed vocabulary of size $V$. Each token receives exactly one integer ID:

For example:

The IDs Are Arbitrary

The sentence I like burgers. might become:

These numbers are identifiers, not quantities. Token 1824 is not larger, more important, or more meaningful than token 17. It is just a label.

Byte Pair Encoding Intuition

Byte Pair Encoding, or BPE, starts with very small pieces and repeatedly merges frequent neighboring pieces. If er appears often, it may become one token. If burger appears often, it may eventually become one token too.

Rare words do not need to become full tokens. They can remain combinations of smaller pieces.

Why Token Count Matters

If a prompt has $n$ tokens, full attention compares token pairs, giving roughly:

More tokens also mean more context-window usage and, in many APIs, more cost. The tokenizer affects computation before the model has even started reasoning.

Why Tokenisation Is Not Enough

Tokenisation gives the model integer IDs. But an ID alone has no semantic meaning:

The model still needs embeddings to turn these arbitrary IDs into vectors that can learn relationships. Tokenisation gives identifiers; embeddings give the model a learnable representation.

In practice, you normally use the tokenizer that belongs to the model. Different models can use different vocabularies and split the same text differently.

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("gpt2")

text = "I would like a cheeseburger."

tokens = tokenizer.tokenize(text)
ids = tokenizer.encode(text)

print(tokens)
print(ids)

# tokenize() returns text pieces.
# encode() returns integer IDs.

The exact output depends on the tokenizer. That is the point: tokenisation is part of the model design.