Sampling: how one continuation is chosen

Section 01

From token scores to one selected token

The model has finished looking at your prompt. It produced logits — raw, unnormalised scores — for every token in its vocabulary. Now it must choose one. This is the decoding step, and it repeats for every token the model generates.

Decoding is where temperature, top-k, and top-p live. These controls reshape the probability distribution before a token is drawn. They affect variance and diversity. They do not supply missing knowledge, establish factuality, or guarantee that two hosted runs produce identical output.

This lesson traces the path from logits to one selected token, then to the next, and the next — because every choice changes the distribution that follows.

Section 02

A forge for probabilities

Think of the decoding step as a forge with adjustable dies. Raw logits enter as ingots of different sizes. A heat dial (temperature) changes how much the ingots soften or harden relative to each other. A cutoff gate (top-k) removes the smallest pieces before they reach the die. A mass threshold (top-p) keeps only enough material to fill the mould with the most likely candidates. One piece is drawn from what remains.

Where the analogy breaks: a real forge is deterministic once set. Decoding involves a random draw — except in greedy mode, which is a forced selection. The forge does not 'decide' which piece is best; it samples from a distribution that the controls have shaped.

Section 03

Temperature-scaled softmax

The decoder divides every logit by the same positive temperature value, then applies softmax to produce probabilities that total 1. Lower temperature widens the gaps between candidates, concentrating probability on the highest-scoring tokens. Higher temperature narrows the gaps, spreading probability across lower-scoring tokens.

A numerically stable implementation subtracts the largest scaled logit before exponentiation. This does not change the resulting probabilities; it prevents numerical overflow.

P(i) = exp(z_i / T) / Σ_j exp(z_j / T)

z_i  = raw logit for candidate token i
T    = positive temperature

Lower T → sharper distribution (favors high scorers)
Higher T → flatter distribution (spreads probability)

Section 04

The Probability Forge

Use the controls below to reshape a live probability distribution. The fixture uses eight candidate tokens with fixed logits — local teaching data, not output from any named provider.

Change one control at a time. If chart motion obscures the comparison, switch to the data table or enable reduced-motion mode. The local RNG seed (2026) makes this fixture repeatable.

Section 05

How the controls combine

The decoder applies controls in a specific order. Temperature scales the logits. Top-k keeps only the k highest-scoring candidates and discards the rest. Top-p (nucleus sampling) sorts the survivors by probability and keeps the smallest prefix whose cumulative mass reaches p. The remaining mass is renormalised to total 1, and one token is drawn.

Greedy mode bypasses all of this: it selects the highest-scoring candidate directly. It is a forced choice, not a probability of 1.

The number of retained candidates changes with the distribution. Top-k fixes the count; top-p adapts it. This is why top-p is called nucleus sampling — the nucleus of most likely candidates grows or shrinks with the shape of the distribution.

function decode(logits, temperature, topK, topP, greedy):
  if greedy:
    return argmax(logits)
  scaled = logits / temperature
  probs = softmax(scaled)
  if topK < len(probs):
    keep top-K candidates by probability
  if topP < 1.0:
    sort by descending probability
    keep smallest prefix with cumulative >= topP
  renormalise remaining probabilities
  return weighted_sample(probs)

Section 06

What decoding cannot do

Greedy decoding removes random sampling from the choice, but it does not search all possible sequences, prove the result globally optimal, or check claims against evidence. Hosted runs may still differ because of model revisions, routing, batching, numerical kernels, or tie-breaking.

Lower temperature reduces variance. It does not supply missing knowledge. A model running at temperature 0 does not become more factual — it becomes more consistent, and consistency is not the same as correctness.

A fixed seed makes this local fixture repeatable, but seeds in hosted APIs are commonly best-effort. Providers may expose different ranges, defaults, clamps, and control order.

Section 07

Unscored lab: choose settings for two tasks

Consider two tasks. The first is JSON extraction: the model must return valid JSON with specific fields from a document. The second is brainstorming: the model should produce diverse, creative continuations from a prompt.

For each task, propose decoding settings (temperature, top-k, top-p). Define what failure looks like — invalid JSON for the first, repetitive or narrow output for the second. Run repeated samples and count failures. Then explain why the final choice depends on measured outcomes, not on a universal recommendation.

1. Propose decoding settings for the JSON extraction task. What temperature? What filters?
2. Define failure metrics: schema violations, missing fields, hallucinated values.
3. Run 20 samples. Count failures. Was the distribution too broad? Too narrow?
4. Propose settings for the brainstorming task. How do they differ?
5. Run 20 samples. Measure diversity: how many distinct continuations?
6. Explain why no universal preset applies to both tasks.