How machines learn from examples

Section 01

The simplest neural network

Before you can understand what an LLM does when you send it a prompt, you need three things: what a neural network is, how it learns, and what a model actually is in code. This lesson builds all three from scratch — no calculus, no linear algebra. If you know what a function is, you know enough.

This lesson ends by bridging to text: text is split into tokens (subword pieces), tokens become numbers, and the model operates on those numbers. That sets up Lesson 00b, where we meet the transformer — the architecture that made modern language models possible.

Section 02

A model is a function with learned parameters

In software engineering, a model usually means a data structure — a schema, a DTO, an ORM entity. In machine learning, a model is something different: a parameterized function. It takes inputs, multiplies them by a set of internal numbers called weights (or parameters), and produces outputs.

Think of it as a machine with thousands of knobs. Each knob is a weight. You feed the machine an input, it produces a prediction, and then you compare the prediction to the correct answer. If it is wrong, you nudge the knobs slightly so that next time the prediction is closer. Do this millions of times across millions of examples, and the machine gets good at the task.

Where the analogy breaks: the knobs are not independent. Each training example nudges many knobs simultaneously, and the interactions between knobs are what give the model its power. It is not tuning one knob at a time — it is tuning thousands at once, where each example tells you a tiny bit about which way to push each one.

Section 03

Layers, activation, and the forward pass

A neural network is not one function — it is a stack of them, called layers. Each layer takes the output of the previous layer, multiplies by its own weights, and passes the result forward. This is called a forward pass: input flows through the layers and an output emerges.

But there is a critical ingredient between each layer: a nonlinear activation function. Without it, stacking linear layers would collapse into a single linear transform — no matter how deep the stack, it could only model linear relationships. The activation function introduces nonlinearity, which is what lets the network model complex, non-linear patterns. Without it, depth buys nothing — a stack of linear transforms collapses to one.

The loss function measures how wrong the prediction was. Backpropagation computes which direction each weight should move; gradient descent applies the nudge. You do not need the math — just the picture: measure the error, compute which way each knob should turn to reduce it, nudge all of them slightly, repeat.

1. Feed input through the layers (forward pass)
2. Compare output to correct answer (loss)
3. Compute which way each weight should move (gradient)
4. Nudge all weights slightly in that direction (update)
5. Repeat with the next example

// After millions of examples, the weights encode
// patterns that produce good predictions.

Section 04

Generalization, not memorization

The goal of training is not to score perfectly on the training data. It is to generalize — to produce good predictions on inputs the model has never seen before. If a model memorizes the training data but fails on new inputs, it has overfit. If it cannot even learn the training data, it has underfit.

This is why you cannot evaluate a model by its performance on training data. You need a separate set of examples the model has not seen — a validation set — to measure whether it has learned useful patterns or merely memorized.

Section 05

From text to numbers: tokens

A neural network operates on numbers, not text. Before any language model can process input, the text must be converted into a format the model can work with. This happens in two steps.

First, text is split into tokens — subword pieces that are not quite words and not quite characters. The word 'tokenization' might become three tokens: 'token', 'iz', 'ation'. Common words are single tokens; rare words break into multiple. The exact splitting depends on the tokenizer, which is specific to each model.

Second, each token is mapped to an integer ID — a number that identifies it in the model's vocabulary. The vocabulary might contain 50,000 to 200,000 tokens. These IDs are what the model actually processes.

This is the bridge to everything that follows: when you send a prompt to an LLM, the API first tokenizes it into IDs, feeds those IDs through the model, and the model produces output token by token. Every concept in this course — context windows, sampling, embeddings, retrieval — operates on tokens, not raw text.

Input:  'The model learns from examples'

Tokens: ['The', ' model', ' learns', ' from', ' examples']
IDs:    [464, 1043, 8452, 422, 4218]

// Different tokenizers split differently.
// IDs are model-specific — no portable meaning.

Section 06

Where learning goes wrong

Overfitting is the most common failure: the model memorizes patterns specific to the training data but does not generalize. The fix is more data, regularization, or early stopping (halting training before the model starts memorizing noise).

Underfitting is the opposite: the model is too simple to capture the patterns. The fix is a larger model, more layers, or longer training.

A subtler failure: the model learns patterns present in the training data that are not true — biases, errors, outdated information. The model has no concept of truth; it learns whatever the data contains. This is why training data quality matters and why models can produce confident, fluent, wrong output.

Section 07

Unscored lab: trace the training loop

No code is required for this lab. The goal is to build the right mental model before you encounter real training code.

1. Pick a simple prediction task: given the number of bedrooms, predict house price. What are the inputs? What is the output?
2. Imagine the model has 10 weights. What does a forward pass look like — how do the inputs flow through to a prediction?
3. The first prediction is random (the weights start random). How do you measure how wrong it is?
4. Describe gradient descent in your own words: how does each weight get nudged?
5. After 10,000 examples, the model is doing well on training data. How do you know if it generalizes?
6. Now extend: instead of predicting house prices, the model predicts the next token in text. What changes? What stays the same?