Long context is not perfect recall

Section 01

Capacity is not capability

Lesson 02 introduced the context window as a capacity limit: the model can accept up to N tokens per request. But there is a difference between accepting tokens and using them well. A model that can receive 128,000 tokens may still fail to find a single fact buried at position 40,000 — not because the fact does not fit, but because retrieval quality varies with position, distractors, and task type.

This lesson is about that gap: between what the context window allows and what the model reliably uses. Understanding it prevents the most common long-context mistake — assuming that if something fits, it will be found.

Section 02

The marked pin among layered material

Imagine a long workbench covered in hundreds of layered sheets of material. Somewhere in those layers, a single pin with a vermilion head is hidden. You have a narrow inspection lens that can examine one section at a time. Your ability to find the pin depends on where it sits, how many similar-looking pins act as distractors, and how carefully you scan.

The model faces a similar problem. Every token in the context window is technically accessible through attention, but attention is not equal across all positions. The model's training distribution, the prompt structure, the density of distractors, and the specific task all affect which tokens the model actually uses when generating its response.

Where the analogy breaks: you move your lens deliberately. The model has no lens — it computes attention weights for all positions simultaneously, and those weights are shaped by the training data and the current input. The model cannot 'try harder' to find a fact; it either attends to it or it does not.

Section 03

Needle-in-a-haystack testing

The needle-in-a-haystack test is the standard diagnostic for position-dependent retrieval. The setup is simple: place a target fact (the needle) at different positions within a long context (the haystack), ask the model to retrieve it, and measure success at each position and context length.

A good result means the model can literally retrieve a fact placed at any position. It does not prove deep comprehension of arbitrary documents. It proves the model can find and reproduce a specific piece of information when asked directly.

By varying context length, needle position, distractor similarity, question wording, and number of needles, you get a retrieval heatmap: a map of where the model is reliable and where it is not. This heatmap is model-specific, task-specific, and prompt-specific. There is no universal shape.

needle = 'The secret code is: vermilion-7421'
question = 'What is the secret code?'

for context_length in [4K, 8K, 16K, 32K, 64K, 128K]:
    haystack = generate_padding(context_length)
    for position in [0%, 25%, 50%, 75%, 100%]:
        doc = insert_at(haystack, needle, position)
        prompt = doc + '\n\n' + question
        response = model.generate(prompt)
        success = evaluate(response, expected='vermilion-7421')
        record(context_length, position, success)

# Plot results as a heatmap: position x length -> success rate

Section 04

Why performance varies by position

Research has shown that some models and tasks exhibit lower performance when relevant information appears in the middle of a long context — a pattern sometimes called 'lost in the middle'. But the shape and severity are model-dependent, task-dependent, prompt-dependent, and data-dependent.

There is no universal U-shaped performance curve. Some models show it strongly; others do not. Some tasks (literal retrieval) are more robust than others (multi-hop reasoning). Prompt structure matters: where you place the question relative to the evidence affects results. Distractor density matters: one needle among dissimilar text is easier than one needle among similar-looking facts.

Section 05

Designing for reliable long-context use

If you cannot rely on uniform retrieval across the context window, you must design around it. Several strategies help:

• Retrieve and rank relevant excerpts rather than dumping entire documents into context.
• Use explicit sectioning with stable identifiers so the model can locate evidence.
• Place the question and required output contract close to the evidence, not far from it.
• Decompose long documents into chunks, process each separately, and aggregate results with verification.
• Repeat critical constraints carefully — but note that duplicated or conflicting instructions can create new failures.
• Evaluate on representative document lengths and target positions before trusting a long-context design.

Section 06

What not to claim about long context

Do not claim that only 10-20% of a context window is usable. This was a rough estimate from early models and does not apply universally. Different models have different effective ranges, and the range depends on the task.

Do not assume that 'beginning and end are remembered; the middle is ignored.' Some models and tasks show position sensitivity; others do not. Measure the actual distribution for your model and your task.

Do not describe long-context performance as a human-like 'attention span.' The model is not paying or losing attention; it is computing weighted relationships that were shaped during training. 'Long-context performance' is the accurate term.

Section 07

Unscored lab: design a long-context evaluation

Design a needle-in-a-haystack evaluation for an incident-response corpus — realistic engineering documents where missing a critical detail has consequences.

1. Define the needle: a specific fact that must be found (e.g., 'the database failed over at 03:47 UTC').
2. Create realistic distractors: similar-looking facts that could be confused with the needle.
3. Choose tested positions: beginning, 25%, 50%, 75%, end.
4. Choose tested context lengths: start with 4K, increase to your model's limit.
5. Define success criteria: exact retrieval? Correct interpretation? Both?
6. Write down what a passing score would still fail to prove. (Literal retrieval does not imply comprehension.)