Tokens and Embeddings: How Language Models Actually Read Text

A language model never sees the sentence you type. It sees a list of integers, and then it sees those integers as lists of decimal numbers. Everything else — the fluent answers, the summaries, the code completion — is built on top of that translation step.

This post walks through that translation with about twenty lines of Python. We will take one short sentence, watch it become numbers, and look at the two different kinds of “embedding” that people talk about, because the word gets used for two genuinely different things.

The examples use Gemma 3 4B from Google and the transformers library from Hugging Face.

Setup

from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("google/gemma-3-4b-it")
tokenizer = AutoTokenizer.from_pretrained("google/gemma-3-4b-it")

Two objects come out of this, and the split between them is the important part:

• The tokenizer turns text into integers. It contains no learned reasoning ability — it is essentially a lookup table plus a splitting algorithm.
• The model turns those integers into vectors and processes them. This is where the billions of learned parameters live.

They are always a matched pair. A tokenizer from one model produces integers that mean something completely different to another model, the way the same phone number reaches different people in different countries.

1: The vocabulary

Every tokenizer has a fixed inventory of pieces it knows how to represent, called the vocabulary.

print(f"Original vocabulary size: {len(tokenizer)}")

Original vocabulary size: 262145

Gemma 3 knows 262,145 distinct tokens. That number sits in an interesting middle ground. English has roughly 170,000 words in current use, plus names, plus other languages, plus code, plus emoji — so 262,145 is nowhere near enough to give every possible word its own slot. But it is far more than the 26 letters you would need for a purely character-by-character approach.

That middle ground is deliberate, and it is called subword tokenization. Common words get their own token. Rare words get assembled from fragments. Nothing is ever truly out-of-vocabulary, because the tokenizer can always fall back to smaller and smaller pieces.

The vocabulary size is not just trivia — it directly sizes part of the model. Gemma 3 4B represents each token as a 2,560-number vector, so its lookup table of token vectors holds 262,145 × 2,560 ≈ 671 million numbers. That is a meaningful share of a 4-billion-parameter model, spent entirely on the dictionary before any actual language processing happens.

2: Text becomes integers

Let’s tokenize a sentence chosen to show a few different behaviors at once.

text = "Hello World. This is tokenization"

tokens = tokenizer(text, return_tensors="pt")
print(tokens)

{'input_ids': tensor([[     2,   9259,   4109, 236761,   1174,    563,   8369,   1854]]),
 'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1]])}

Two things came back.

input_ids is the sentence, now a list of eight integers. This is the only form of the text the model will ever handle. The nesting matters: the shape is [1, 8], meaning one sequence of eight tokens. That leading 1 is the batch dimension, and it is there because models are built to process many sequences at once for efficiency, even when you only hand them one.

attention_mask is a row of eight ones, telling the model that all eight positions are real content. It looks pointless here, and for a single sentence it is. It earns its place when you batch sentences of different lengths together: shorter ones get padded to match the longest, and the mask marks those padding positions with 0 so the model learns to ignore them rather than treating filler as meaning.

3: Decoding, one token at a time

The integers are more interesting when we translate each one back:

for id in tokens['input_ids'][0]:
    print(tokenizer.decode(id))

<bos>
Hello
 World
.
 This
 is 
 token
ization

Five words became eight tokens. Each difference between the two teaches something about how tokenizers work.

<bos> appeared out of nowhere. This is a special token meaning “beginning of sequence,” and the tokenizer inserted it automatically. It corresponds to no text you wrote. Special tokens are control signals — markers for the start of input, the end of a turn, the boundary between a system instruction and a user message. If you have ever wondered how a chat model knows where your message stops and its own reply should start, the answer is tokens like this one.

Spaces live inside the tokens. Look closely at Hello versus ` World. The second has a leading space; the first does not. The tokenizer treats "a space followed by World" as a single unit, which is why World at the start of a sentence and World` mid-sentence are different token IDs. This is not a quirk to work around — it is how the tokenizer represents word boundaries without wasting a separate token on every space.

Punctuation stands alone. The period became its own token, 236761, rather than attaching to World. This keeps World and World. from being two unrelated entries in the vocabulary.

“tokenization” split in half. This is the most revealing line in the output. The word became ` token + ization`, two tokens. The word “tokenization” was not frequent enough in the training data to earn a dedicated slot, so it gets assembled from a common word and a common suffix.

That last point has real consequences. A model does not perceive “tokenization” as one indivisible concept; it perceives a familiar stem carrying a familiar grammatical ending. This is a large part of why models handle unfamiliar technical vocabulary and made-up words gracefully rather than failing outright. It is also why they are famously bad at counting the letters in a word — the letters were never individually visible.

The practical version of this: token count is not word count. Since API pricing, context limits, and processing cost are all measured in tokens, and since English text runs roughly 1.3 tokens per word (with code, unusual names, and non-English text running higher), a document is usually more expensive than its word count suggests.

4: Integers become vectors

Now the model itself. We hand it the tokenized input and look at what comes back.

output = model(**tokens)[0]
output.shape

torch.Size([1, 8, 2560])

Read that shape as: 1 sequence, 8 tokens, 2,560 numbers per token.

Each token has been turned into a list of 2,560 decimal numbers — a vector. This is a token embedding, and it is the model’s internal representation of that piece of text. The count of 2,560 is a fixed architectural choice for Gemma 3 4B; larger models use wider vectors.

The reason for this format is that integers make terrible inputs for arithmetic on meaning. Token 9259 is not “less than” or “half of” token 4109 in any sense that matters — the IDs are arbitrary labels, like employee numbers. Vectors, by contrast, live in a space where distance is meaningful. Vectors for related concepts sit near each other, and the model learned that arrangement from its training data. All of the mathematics inside the model operates on these vectors, never on the raw IDs.

The crucial detail is that these vectors are contextual. The vector for a token is not a fixed dictionary entry; it is computed by the model in light of the surrounding tokens. Feed in “river bank” and “savings bank,” and the vector for bank differs between them, because the model has looked at the neighboring words. This is the central advance of modern transformer models over older embedding methods, which gave every occurrence of a word the same vector regardless of context.

Notice also that the output has one vector per token. Eight tokens in, eight vectors out. There is no single vector here representing the sentence as a whole — which brings us to the second kind of embedding.

5: One vector for a whole sentence

Often you don’t want eight vectors. For search, clustering, recommendation, or duplicate detection, you want one vector standing for an entire sentence, paragraph, or document, so you can compare texts directly.

That is a different tool, from the sentence-transformers library:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer('sentence-transformers/all-mpnet-base-v2')

vector = model.encode(text)
vector.shape

(768,)

One vector. 768 numbers. No token dimension at all.

Compare the two shapes side by side, because this is the distinction the word “embedding” hides:

The text embedding is fixed size regardless of input length. A five-word sentence and a five-paragraph document both produce exactly 768 numbers. That fixed size is the entire point: it makes texts of any length directly comparable. You can measure the distance between two of these vectors and get a usable answer to “how similar in meaning are these two documents?”

This is the machinery behind semantic search, and behind retrieval-augmented generation (RAG). Embed your documents once, store the vectors, and at query time embed the question and find the nearest document vectors. The search matches meaning rather than keywords, which is why it can connect a question about “car maintenance” to a document about “automobile repair” with no shared words.

A note on the code above: it reuses the variable name model, replacing the Gemma model. That is fine when running cells top to bottom, but in a real script give them distinct names — the two objects have different interfaces and are not interchangeable.

Summary

The path from text to something a neural network can compute on has three steps:

1. Tokenization splits text into subword pieces from a fixed vocabulary and maps each to an integer. Gemma 3’s vocabulary holds 262,145 entries. Common words stay whole, rarer words split into fragments, and special tokens like <bos> mark structure.
2. Token embeddings convert each integer into a vector — 2,560 numbers per token for Gemma 3 4B. These are contextual: the same word gets a different vector depending on the words around it. You get one vector per token.
3. Text embeddings compress an entire passage into a single fixed-size vector — 768 numbers for all-mpnet-base-v2 — so that texts of different lengths can be compared. This is what powers semantic search and RAG.

The distinction in those last two points is worth carrying with you, because both are called “embeddings” and they solve different problems. If you are generating text, you want one vector per token. If you are comparing or retrieving documents, you want one vector per document.

Everything a language model does with language, it does with these numbers. Understanding the translation makes the rest of the system considerably less mysterious.