The transformer insight

Section 01

From single inputs to sequences

Lesson 00a covered neural networks that process individual inputs. But language is a sequence — the meaning of each word depends on the others. 'Dog bites man' and 'man bites dog' contain the same words in different order, and the meaning is completely different.

Before 2017, models that processed sequences mostly did it one element at a time. The transformer architecture changed that: it processes all elements simultaneously, looking at every relationship at once. This is called attention, and it is the single insight that made modern language models possible.

Section 02

Reading a sentence all at once

Imagine reading the sentence: 'The bank was steep and slippery.' When you read 'bank', you do not know if it is a river bank or a financial bank — until you read 'steep'. The meaning of 'bank' depends on its relationship to other words.

A model processing one word at a time would have to decide what 'bank' means before seeing 'steep'. It would guess wrong. A transformer processes all tokens at once, letting each token attend to the tokens before it and decide which ones are relevant. In the decoder-only models this course covers, each token can only see prior tokens — not future ones. The word 'bank' pays attention to 'steep' and 'slippery' and its representation resolves toward the river-bank sense.

The key insight: which connections matter is itself learned. The model is not told that 'bank' should attend to 'steep' — it learns this from billions of examples during training. The attention weights are dynamic; they change based on the content and context of each specific input.

Section 03

Embeddings, attention, and positional encoding

Before a transformer can process text, token IDs must become embeddings. An embedding is a vector — a list of hundreds or thousands of numbers — where tokens with similar meanings are mapped to nearby points in the vector space. 'cat' and 'dog' would be close; 'cat' and 'airplane' would be far apart. The embeddings are learned during training.

Attention works on these embeddings. For each token, the transformer computes a weight for every other token: how much should this token pay attention to that one? These weights are computed from the embeddings using learned matrices — three sets of weights called query, key, and value. The query of one token is compared to the key of every other token; the comparison determines the attention weight.

Positional encoding solves a specific problem: attention is permutation-equivariant by default. Shuffle the input and the output shuffles with it — the relationships are unchanged, because attention has no sense of order — because it only looks at relationships, not order. But order matters in language. Positional encoding adds position information to each embedding before attention, so the model knows which token came first, second, and so on.

1. Embed:   token IDs → vectors (embeddings)
            + add position information

2. Attend:  each token computes attention weights
            against every other token
            (query × key → weight; weight × value → output)

3. Repeat:  stack multiple attention layers,
            each learning different relationships

// The output is a new set of vectors where each
// token's representation now incorporates context
// from all other tokens.

Section 04

Stacked layers: deeper attention

A transformer is not one attention step — it is many, stacked. Each layer takes the output of the previous layer and applies attention again. The first layer might learn syntactic relationships (subject-verb). Deeper layers might learn semantic relationships (anaphora resolution, negation scope). The deepest layers might learn task-level reasoning patterns.

This stacking is why 'deep' learning matters: each layer builds on the abstractions learned by the previous one. A 96-layer transformer sees relationships a 2-layer transformer cannot.

'Multi-head attention' means the model runs multiple attention computations in parallel at each layer, each looking for different kinds of relationships. One head might focus on local syntax; another might track long-range dependencies. The results are combined before passing to the next layer.

Section 05

Why transformers replaced RNNs

Before transformers, sequence models were mostly recurrent neural networks (RNNs) and their gated variants, like LSTMs. These processed tokens one at a time, passing a hidden state forward. The problem: by the time the model reached token 500, the signal from token 1 had decayed. Long-range dependencies were hard to learn.

Transformers solved this in two ways. First, attention gives every token direct access to every other token — no information needs to survive a chain of hidden states. Second, because attention is computed in parallel (not sequentially), transformers can be trained much more efficiently on GPUs. The parallelism is not just faster — it makes training on massive datasets economically feasible, which is a prerequisite for the scaling that produces capable models.

Section 06

What attention does not solve

Attention gives every token access to every other token, but it does not give unlimited capacity. A fixed number of layers and heads means the model can only learn so many relationship patterns. More layers and heads help, but at the cost of more parameters and compute.

The attention computation scales quadratically with sequence length: doubling the input quadruples the number of attention comparisons. This is why context windows have a hard limit — it is not just memory, it is computational. Many optimizations (sparse attention, sliding window, flash attention) reduce this cost, but the quadratic baseline remains a fundamental constraint.

Section 07

Unscored lab: trace a sentence through attention

No code required. Build the mental model first; the mechanics will make more sense when you see them in Lesson 01.

1. Take the sentence: 'The crane lifted the heavy steel beam.' The word 'crane' is ambiguous (bird or machine?). Which other words resolve the ambiguity?
2. If you were designing attention weights for 'crane', which tokens should get high weights? Which should get low weights?
3. Now consider: the model does not know English grammar. It must learn these weights from data. What kind of training examples would teach it that 'crane' near 'lifted' and 'beam' means a machine?
4. Imagine 96 layers of attention stacked. What might the first layer learn? What might layer 50 learn? What about layer 95?
5. Why does positional encoding matter? Without it, what would the model fail to distinguish?