From architecture to LLM

Section 01

The core objective: predict the next token

You now know what a neural network is (Lesson 00a) and what the transformer architecture does (Lesson 00b). Now we connect them to the thing you actually use: a large language model.

An LLM is trained on one task: given a sequence of tokens, predict what token comes next. That is it. The model sees billions of text examples — web pages, books, code, conversations — and for each, it tries to predict the next token given everything before it. Over trillions of tokens, this simple objective produces a model that can generate coherent text, answer questions, write code, and reason about problems.

Everything that makes an LLM useful beyond raw text generation — instruction following, helpful responses, safety — comes from additional training stages applied after pretraining. We will trace the full pipeline now.

Section 02

The apprentice's journey

Think of the LLM pipeline as the training of an apprentice craftsman.

First, pretraining: the apprentice reads everything in the workshop's library — books, manuals, blueprints, letters. They learn how language works, how ideas connect, how to form coherent sentences. They can now generate plausible text, but they do not yet know how to be helpful. Ask them a question and they might continue the text rather than answer it.

Then, instruction tuning: the apprentice is given examples of good task performance — here is a question, here is a good answer. They learn to follow instructions, not just continue text. Now when you ask a question, they answer instead of generating more questions.

Finally, preference optimization: the apprentice is shown two responses to the same prompt and told which one the master prefers. Over many comparisons, they develop judgement about quality — not just 'is this a valid response' but 'is this the better response.'

Where the analogy breaks: scaling (more parameters, data, compute producing more capability) has no clean counterpart in apprenticeship. A human apprentice does not suddenly get better because their brain got bigger. Model scaling produces capabilities that may or may not be genuine emergence — how much is real versus an artifact of how we measure is actively debated.

Section 03

The training pipeline

Pretraining: The model is trained on massive text corpora with the next-token prediction objective. Given a sequence of tokens, it predicts the next one; the loss measures how wrong the prediction was; gradient descent nudges the weights. This stage consumes the vast majority of compute — months on thousands of GPUs. The output is a base model: it can generate fluent text but does not follow instructions reliably.

Instruction tuning (also called supervised fine-tuning): The base model is further trained on curated examples of instruction-response pairs. This teaches it to recognize instruction format and respond appropriately rather than just continuing text. Safety considerations are typically woven into this stage and the next, not applied as a separate clean phase.

Preference optimization (RLHF or DPO): The model is given pairs of responses to the same prompt. Humans (or other models acting as judges) rate which response is better. The model learns to prefer responses that humans rate highly. RLHF (Reinforcement Learning from Human Feedback) and DPO (Direct Preference Optimization) are two methods for this; the details vary, but the goal is the same: align the model's output distribution with human preferences.

1. PRETRAINING
   Objective: predict next token from trillions of tokens
   Output: base model (fluent but not instruction-following)
   Cost:   months on thousands of GPUs

2. INSTRUCTION TUNING (supervised fine-tuning)
   Input:  curated instruction-response pairs
   Output: model that follows instructions
   Safety considerations begin here

3. PREFERENCE OPTIMIZATION (RLHF or DPO)
   Input:  pairs of responses, rated by preference
   Output: model aligned with human judgement
   RLHF: reinforcement learning on reward model
   DPO:  direct optimization on preference data

Section 04

Scaling: more parameters, data, compute

The transition from 'a transformer model' to 'a large language model' is primarily about scale. Modern LLMs have billions to trillions of parameters. They are trained on trillions of tokens. They run on clusters of specialized hardware — GPUs or TPUs designed for the massive parallel matrix operations that attention and feedforward layers require.

More scale generally produces more capability: better language understanding, better reasoning, better instruction following. This is not a smooth curve. Some capabilities appear to grow sharply at certain thresholds — researchers call this 'emergence'. But how much is genuine versus an artifact of how we measure is debated. The evaluation metrics we use may make gradual improvement look sudden.

What is not debated: larger models trained on more data do better on practically every benchmark. The scaling laws are empirical and remarkably consistent across model families.

Section 05

The context window

A trained LLM has a context window: a fixed maximum number of tokens it can process in a single forward pass. This is a hard architectural limit, not a suggestion. If the input exceeds it, the model cannot process all tokens at once.

The context window is set by the model's architecture (how many positions the positional encoding supports) and training (the maximum sequence length seen during training). A model trained on 4,096-token sequences may not perform well on 32,000-token inputs even if the architecture technically allows it.

Context windows have grown dramatically: from 2K tokens in early GPT-3 to 128K-1M+ in current models. But a larger context window does not guarantee equally reliable use of every token within it — Lesson 03 explores this distinction in depth.

Section 06

What the pipeline does not guarantee

Pretraining produces a model that generates fluent text. Fluency is not truth. The model learned to predict plausible continuations, not to verify facts. It will confidently generate text that reads well and is wrong.

Instruction tuning produces a model that follows instructions. It does not produce a model that knows when to refuse, when to ask for clarification, or when its knowledge is outdated. Those behaviors require additional training and careful system design.

Preference optimization produces a model that generates responses humans rate highly. Human preferences are not the same as correctness. A model optimized for human approval may learn to produce confident-sounding answers rather than hedged, uncertain ones — because humans prefer confidence.

Section 07

Unscored lab: trace a real prompt through the pipeline

This lab connects Module 0 to Module I. Everything in Module I builds on the mental model you construct here.

1. Write a simple prompt: 'Explain what a cache is to a junior developer.'
2. Trace it: the prompt is tokenized into IDs. How many tokens might this be?
3. The token IDs become embeddings. What does each embedding represent?
4. The transformer layers apply attention. For the word 'cache', which other tokens should receive high attention?
5. The model produces logits — scores for the next token. What are the most likely next tokens?
6. The decoder selects one token. What controls that selection? (You will learn this in Lesson 05.)
7. The selected token is appended, and the process repeats. How long does this go on?
8. Now you are ready for Module I: trace the full path from raw text to the next token.