From text to the next token

Section 01

What happens when you press send

In Module 0, you saw that an LLM is a transformer model trained to predict the next token. Now let's trace exactly what happens when you send it a prompt — from the moment your text enters the API to the moment a generated token appears.

The path has five stages: tokenization (text becomes token IDs), embedding (IDs become vectors), the forward pass (transformer layers combine information), decoding (scores become one selected token), and autoregressive repetition (the selected token is appended and the process repeats).

Understanding this path is the foundation for everything else in this course. Token usage, latency, tool calls, unexpected output — all of it traces back to these stages.

Section 02

The cutting room

Think of tokenization as a cutting room. Your prompt — a continuous stream of text — is fed into a precision cutter that slices it into discrete pieces. The cuts are not at word boundaries; they follow a vocabulary learned during the model's training. Common words are single pieces. Rare words break into multiple. Code, whitespace, emoji, and non-English text all cut differently.

Each piece is then stamped with a number — its token ID. These IDs are model-specific. Token 464 in one model might be 'The'; in another model, it might be something entirely different. The IDs carry no portable meaning. They are lookup keys into the model's embedding table.

Where the analogy breaks: the cutter is not random, but it is not intuitive either. You cannot predict token counts from character counts. Four English characters per token is a rough heuristic that breaks for code, Japanese, whitespace, and subword boundaries. The only reliable way to count tokens is with the actual tokenizer.

Section 03

The forward pass: from IDs to logits

Once text is tokenized into IDs, the forward pass begins. Three things happen:

First, each token ID is mapped to its embedding — a learned vector where the model's training has encoded relationships between tokens. This happens via a simple lookup table: ID 464 maps to a specific vector of (typically) thousands of dimensions.

Second, positional information is added to each embedding. As you saw in Lesson 00b, attention is permutation-invariant by default. Positional encoding tells the model where each token sits in the sequence.

Third, the transformer layers process these positioned embeddings through attention and feedforward networks. Each layer combines information from all previous tokens under the model's attention rules. The output of the final layer is projected to produce logits: unnormalised scores, one for every token in the vocabulary. These logits are not probabilities — they are relative scores that the decoder will transform.

# 1. Tokenize
token_ids = tokenizer.encode(prompt)

# 2. Embed + add position
embeddings = embedding_table[token_ids]
embeddings += positional_encoding(len(token_ids))

# 3. Transformer layers
for layer in transformer_layers:
    embeddings = layer(embeddings)  # attention + feedforward

# 4. Project to vocabulary
logits = output_projection(embeddings[-1])  # scores for next token

# 5. Decode (Lesson 05 covers this in detail)
next_token = decode(logits)
token_ids.append(next_token)
# repeat from step 2

Section 04

Autoregressive generation: one token at a time

Generation is not a single forward pass that produces an entire response. It is a loop: the model produces logits for one next token, the decoder selects one, that token is appended to the sequence, and the entire forward pass runs again with the longer sequence.

This is called autoregressive generation — 'auto' (self) + 'regressive' (from previous values). Each token is predicted from all tokens that came before it. The model cannot skip ahead or plan multiple tokens in advance within a single decoding step.

Small differences compound. Because each selected token becomes part of the input for the next step, a different selection at step 1 changes the distribution at step 2, which changes step 3, and so on. This is why the same prompt at different temperatures can produce wildly different outputs — not because the model changed, but because the first divergent token cascaded.

Section 05

The KV cache: reusing computed state

During generation, each transformer layer computes attention over all previous tokens. Without optimization, step 500 would recompute attention for all 499 previous tokens — work that was already done at step 499.

The KV cache solves this. At each decoding step, each layer retains the key and value tensors computed for prior tokens. When the next token is processed, only the new token's key and value are computed; the previous ones are reused from the cache. This significantly improves decode latency for long sequences.

The tradeoff is memory. The cache grows with sequence length, number of layers, number of attention heads, and precision. A long conversation can consume substantial GPU memory just for cached attention state.

Important boundaries: the KV cache is temporary inference state. It is not durable memory. It is not a vector database. It is not 'prompt caching' (which is a provider billing optimization). And it does not expand the model's context window — it only avoids recomputing tokens that already fit.

Section 06

Common misconceptions

The model is not searching a database of memorized sentences. During generation, it computes a probability distribution over the vocabulary and samples from it. The training data influenced the weights, but the weights are not a lookup table of stored text.

A token is not the same as a word. Depending on the tokenizer, one word may be multiple tokens, and one token may span multiple characters. API billing, context limits, and latency all operate on token counts — not word counts.

Fluent output is not evidence of understanding. The model produces tokens that are statistically likely given the training data and the current context. Fluency means the model learned the statistical patterns of language well — it does not mean the model verifies, fact-checks, or comprehends.

Section 07

Unscored lab: instrument a toy chat request

Using any LLM API (or the provider documentation), instrument a simple chat request and record what you observe.

1. Send a short prompt. Record: input token count, output token count, time to first token, per-token decode time, and stop reason.
2. Compare input token count to character count. What is the ratio? Try the same prompt with code, emoji, or non-English text.
3. Send the same prompt twice. Are the outputs identical? Why or why not? (Hint: what controls the selection?)
4. Record latency. Is time-to-first-token different from per-token decode time? What does this tell you about prefill vs decode?
5. Identify which measurements belong to the model provider and which belong to your application code.