Context is a finite workspace

Section 01

The bounded workbench

In Lesson 01, you traced a prompt through tokenization, the forward pass, and autoregressive generation. Now consider: the model can only process a fixed number of tokens in one request. This is the context window — a hard limit on what fits on the workbench at once.

Everything the model needs for a single inference must fit within this window: system instructions, tool definitions, conversation history, retrieved documents, the current user input, tool results, and reserved space for the output the model will generate. If the total exceeds the limit, something must go.

Managing this budget is not a detail — it is a core systems design problem. How you allocate the context window directly affects what the model can do, what it remembers, and how reliably it performs.

Section 02

Packing a finite tray

Think of the context window as a workbench tray with fixed dimensions. You have system instructions (the rules of the workshop), tool schemas (the available instruments), conversation history (notes from previous exchanges), retrieved evidence (reference pages pulled from the archive), the current input (the task at hand), and reserved output space (room for the model to work).

Every component competes for the same finite space. A long conversation history might leave no room for retrieved evidence. Detailed tool schemas might consume the budget needed for the user's actual request. A large retrieved document might crowd out the system instructions that define how the model should behave.

Unlike a physical tray, the context window has no concept of priority. The model processes all tokens equally — it does not know that your system instructions are more important than old conversation turns. If you want something protected, you must enforce that in application code.

Section 03

What consumes the budget

The context budget is consumed by:

• System and developer instructions — the rules and role definitions.
• Tool definitions and schemas — JSON Schema descriptions of available tools.
• Conversation history — all prior turns, including assistant responses.
• Retrieved documents and metadata — RAG context, search results.
• Current user input — the actual request.
• Tool results — output from previous tool calls in this conversation.
• Reserved output capacity — space for generated tokens.

Section 04

Tokens are not a fixed character conversion

'About four English characters per token' is a rough heuristic that works for clean English prose and nothing else. Token density changes with language, code, whitespace, numbers, emoji, and tokenizer version.

Japanese text may produce 2-3 tokens per character. Python code may produce more tokens than equivalent English prose due to operators, indentation, and identifiers. JSON with nested structure consumes more tokens than the data it represents. Whitespace — leading spaces, blank lines, indentation — is tokenized too.

The only reliable way to count tokens is with the actual tokenizer or the provider's usage metadata. Estimating from character counts leads to silent truncation, rejected requests, and budget calculations that are wrong by 20-40%.

'Hello world'           → 2 tokens  (clean English)
'こんにちは世界'          → ~4 tokens (Japanese)
'def fib(n): return n if n<2 else fib(n-1)+fib(n-2)'
                        → ~25 tokens (Python code)
'{"name":"value","arr":[1,2,3]}'
                        → ~15 tokens (JSON)

// Always count with the real tokenizer.
// These numbers vary by model and tokenizer version.

Section 05

Overflow is an application decision

When a request exceeds the context limit, what happens? It depends on who is handling it.

The model provider may reject the request outright — returning an error, not a truncated prompt. A provider SDK or application layer may trim, compact, summarize, or omit content before sending. The model itself does not silently drop the oldest turns — that is a common misconception. What gets removed, and in what order, is an application policy, not a model behavior.

If you are building the application, you own this decision. You must define: what is protected (system instructions usually), what is trimmed first (old conversation turns, verbose tool output), what is summarized (long retrieved documents), and what triggers each strategy.

Section 06

Compaction strategies and their risks

When the context budget overflows, several strategies exist — each with tradeoffs:

• Remove irrelevant turns: discard conversation history that is no longer needed. Risk: the model loses context it needed.
• Summarize prior conversation: replace old turns with a compressed summary. Risk: summary loses nuance or introduces errors.
• Store durable state externally: keep facts in a database, retrieve only what is needed. Risk: retrieval quality determines what the model knows.
• Use sliding windows: keep only the most recent N turns. Risk: important earlier context is lost.
• Retrieve focused chunks: instead of entire documents, retrieve smaller relevant sections. Risk: relevant information is split across chunks.

Section 07

Unscored lab: design a context budget

Given a 32K context limit and a long customer support conversation, design a budget management plan.

1. List all components that consume tokens: system instructions, tool schemas, history, current input, output reserve.
2. Assign budgets to each component. Which gets the most? Which is most constrained?
3. Define what triggers compaction: what threshold? What gets trimmed first?
4. Design a summary-based plan: what goes into the summary? What is protected?
5. Design a retrieval-based plan: what gets stored externally? What gets retrieved per request?
6. Compare: which plan preserves more useful context? Which is simpler to implement? Which fails more gracefully?