Context windows: the only door into a frozen model

We ended the Substrate section with a single sentence to carry forward: an LLM is a frozen function over tokens with no live access to the world. The weights are read-only after training (the training post). Inference only reads them (the inference post). The knowledge cutoff is baked in at training time (the knowledge cutoffs post). So here's the obvious follow-up, the question that opens this whole Interface section: if the model is frozen, how does anything new ever get in?

Through exactly one door. The context window.

Every fact the model didn't memorize during training — today's date, the contents of the file you just pasted, the result of a tool call, the last thing you said — reaches the model only by being turned into tokens and placed in the window. The context window is the model's working memory, and it's the only write surface a frozen function has. Get this concept right and the rest of the Interface section — roles, sampling, streaming, structured output — is just detail about how you use that door well.

This post is about three things: what the window actually is, why it's bounded, and the gap between the size on the spec sheet and the size you can actually use.

What the window actually is

The context window is the full sequence of tokens the model processes in a single forward pass. That's it. It is not "your prompt" — your prompt is one part of it. In a real application the window holds, all at once:

• the system prompt,
• the conversation history (every prior user and assistant turn),
• any retrieved documents (RAG),
• tool/function definitions,
• tool call outputs, and
• the model's own generation so far — each token it writes becomes part of the context for the next token (that's just autoregressive decoding from the inference post).

All of these draw from one shared budget, measured in tokens. (If you need a refresher on what a token is, see the tokens post — we won't redefine it here.) There is no separate "memory" the model consults on the side. When people say a model "remembers" the conversation, what's actually happening is that the entire history is re-fed into the window on every turn. Nothing persists in the weights between turns, or between chats. Close the session and the model is exactly as frozen and ignorant as it was before — the next session rebuilds the window from scratch.

This is the first misconception to kill: a bigger context window ≠ more memory in the human sense. It's a bigger desk, wiped clean every time you sit down, not a bigger brain.

Why it's bounded

If the window is so central, why not make it infinite? Three mechanical reasons, and they're worth naming precisely.

1. Attention scales roughly quadratically. In the standard transformer, every token attends to every prior token. Double the sequence length and you roughly quadruple the pairwise relationships the model computes during prefill (the one-shot pass over your prompt, from the inference post). This is the famous O(n²) cost. So "doubling the context doubles the compute" is wrong — it's worse than linear. This is why a long prompt is slow before the model has written a single token.

2. The KV cache grows linearly and lives in memory. Recall from the inference post that the KV cache stores the per-token, per-layer key and value tensors so decode doesn't recompute the whole sequence each step. That cache's size scales linearly with sequence length × number of layers × hidden size × batch size. For a 100K-token prompt this is on the order of hundreds of megabytes to a gigabyte of GPU memory, per request, held for the life of the session. Long contexts aren't just slow; they're expensive to hold. Memory pressure, not the raw math, is often the real serving constraint.

3. Positional encoding has a trained range. A transformer needs to know token order, and modern models inject that with Rotary Position Embeddings (RoPE, Su et al., 2021), which largely replaced earlier sinusoidal and learned schemes; some models use alternatives like ALiBi (Press et al., 2021). The catch: the model only saw positions up to some length during training. Push beyond that range and quality degrades, because the model is being asked to interpret position signals it never learned to read. A whole family of techniques — Position Interpolation (Chen et al., 2023), NTK-aware scaling, and YaRN (Peng et al., 2023) — exists to stretch a model trained at, say, 32K out to 128K or beyond by rescaling those position signals, usually with some fine-tuning. The takeaway, without the math: a model's advertised length is often an extended length, not the length it was natively trained at. Llama 4 Scout, for instance, advertises a 10-million-token window but was pre- and post-trained at 256K, relying on architectural tricks (interleaved attention layers, "iRoPE") to generalize beyond that.

Hold onto that last point — it's the seam where the spec sheet and reality come apart.

The size race

Context windows have grown astonishingly fast:

• ~2K tokens — GPT-2 and GPT-3 era (2019–2020).
• 4K–16K — GPT-3.5; Llama 1 (2K) and Llama 2 (4K).
• 128K — GPT-4 Turbo; Llama 3.1.
• 200K — Claude 2.1 (November 2023), which Anthropic described as "roughly 150,000 words, or over 500 pages of material."
• 1M+ — Gemini 1.5 Pro (February 2024, with up to 2M in preview); GPT-4.1 (April 2025, "up to 1 million tokens of context—up from 128,000 for previous GPT-4o models"); Gemini 2.5 Pro ("a 1 million token context window (2 million coming soon)").
• 10M — Llama 4 Scout (April 2025), which Meta called "an industry-leading context window of 10M."

As of mid-2026 the frontier is roughly: Claude models default to 200K with a 1M beta on select models; Google's Gemini Pro line offers 1M (with 2M tiers); OpenAI's GPT-5 family ships around 400K, with later revisions exposing a 1M API window; open models like DeepSeek-V3 sit at 128K. These numbers move monthly, so treat any specific figure as a snapshot, not a constant — and always check the vendor's current model page, because the same model name can expose different limits on different platforms (Azure deployments of GPT-4.1, for example, have shipped capped at 128K despite the model's 1M spec).

A million tokens is genuinely useful: it's the difference between RAG-or-bust and "just paste the whole codebase." But the headline number invites the most expensive misconception in the field.

Advertised ≠ effective: the part that actually bites

Here is the surprise that matters most for engineers: the context length on the model card is a max-rated spec, not a sweet spot. It's the car's top speed, not the speed you want to cruise at. The model will accept that many tokens without erroring; it will not necessarily use them well.

The evidence comes in three escalating tiers of difficulty.

Tier 1 — Needle in a haystack (easy mode). The original test, popularized by Greg Kamradt in late 2023, hides a single out-of-place sentence — "The best thing to do in San Francisco is eat a sandwich and sit in Dolores Park on a sunny day" — somewhere in a long pile of Paul Graham essays and asks the model to retrieve it. Early results already showed cracks: Kamradt found Claude 2.1's recall degrading near the bottom of long documents starting around 90K tokens, and GPT-4 Turbo's around 65K. Today's frontier models mostly ace single-needle retrieval; Google reports Gemini at 99.7% recall at 1M tokens on its own benchmark, and OpenAI says GPT-4.1 "maintains strong performance even up to 1 million tokens" on retrieval. But — and this is the trap — near-perfect needle scores are precisely why people assume the whole window is usable. Single-needle retrieval is the easiest possible long-context task, and as Chroma notes, "NIAH is fundamentally a simple retrieval task."

Tier 2 — Lost in the middle (position bias). The seminal finding here is Liu et al., "Lost in the Middle: How Language Models Use Long Contexts" (arXiv:2307.03172, TACL 2024). Across multi-document QA and key-value retrieval, they found a robust U-shaped performance curve. In their own words: "performance is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts, even for explicitly long-context models." The magnitude is striking — GPT-3.5-Turbo's multi-document QA accuracy swung by more than 20 points depending only on where the answer sat (best-case ~75.8% with the answer at the start vs. ~53.8% in the middle of a 20-document context). And in the worst case its middle-of-context performance fell below its closed-book baseline: per the paper's Table 1, GPT-3.5-Turbo scores 56.1% with no documents at all and 88.3% with only the correct ("oracle") document — yet burying the right document in the middle dragged accuracy under that 56.1% closed-book number. Giving the model the answer in the wrong place made it do worse than giving it nothing. The U-curve has held up across architectures and is one of the most replicated results in the area.

Tier 3 — Hard, realistic tasks (the real ceiling). NVIDIA's RULER benchmark (Hsieh et al., 2024) goes beyond single needles to multi-needle retrieval, multi-hop tracing, aggregation, and QA. Its central concept is effective length: the longest input at which a model stays above a quality threshold, defined as Llama-2-7B's performance at 4K (85.6%). The findings are sobering. NVIDIA reports that of models claiming 32K or greater, "only half of them can effectively handle sequence length of 32K by exceeding [the] qualitative threshold," and "almost all models fall below the threshold before reaching the claimed context lengths." Concretely: GPT-4-1106-preview claimed 128K but had an effective length of 64K (its score falls to 81.2 at 128K); Llama 3.1 70B, also 128K claimed, likewise effective at 64K (dropping to 66.6 at 128K). The pattern: effective context is routinely half of advertised context, or less, on hard tasks.

And it keeps being confirmed. Chroma Research's July 2025 report "Context Rot: How Increasing Input Tokens Impacts LLM Performance" (Kelly Hong, Anton Troynikov, Jeff Huber) tested 18 current models — including GPT-4.1, Claude 4, Gemini 2.5, and Qwen3 — and concluded: "models do not use their context uniformly; instead, their performance grows increasingly unreliable as input length grows." Million-token windows move where the degradation starts; they don't abolish it.

The honest summary is neither "long context is solved" nor "long context is fake." It's this: retrieval of one clear fact works well; synthesis across a long, noisy context is much harder, degrades with length, and is sensitive to where information sits. Bigger windows give you capacity, not quality.

Context engineering: the actual job

If the window is a finite, shared, quality-sensitive budget, then the central skill is deciding what to spend it on. That skill has a name that gained real traction in 2025: context engineering. Shopify's CEO Tobi Lütke kicked it off (X, June 18, 2025) — "the art of providing all the context for the task to be plausibly solvable by the LLM" — and Andrej Karpathy endorsed it a week later (June 25, 2025): "+1 for 'context engineering' over 'prompt engineering'… in every industrial-strength LLM app, context engineering is the delicate art and science of filling the context window with just the right information for the next step." His mental model is one engineers feel in their bones: the LLM is the CPU, the context window is the RAM, and you are the operating system deciding what to page in. Anthropic later formalized it as "the set of strategies for curating and maintaining the optimal set of tokens (information) during LLM inference," explicitly framing prompt engineering as a subset of context engineering.

Concretely, you are allocating one budget across: system prompt, few-shot examples, conversation history, retrieved knowledge, tool definitions, tool outputs, and the model's own output. Every token you spend on one is a token unavailable to the others — and past some point, an extra token can actively hurt by pushing the signal into the lost-in-the-middle zone or simply adding noise.

The trade-offs are concrete:

• Latency — a longer prompt means a bigger prefill, which is the quadratic-ish cost paid before the first output token appears.
• Cost — you pay per token, every turn; a fat system prompt re-sent on every request adds up fast.
• Quality — counterintuitively, more context can mean worse answers (context rot, position bias). Anthropic's own guidance treats context as "a finite resource with diminishing marginal returns." "Just stuff everything in" stops being a strategy past a certain scale.

This is also where prompt caching earns its place — and it connects straight back to the KV cache. When you send the same long prefix repeatedly (a big system prompt, a fixed document), the provider can store the already-computed KV tensors and reuse them instead of recomputing prefill. Anthropic's prompt caching (launched August 2024) is developer-controlled via a cache_control marker, with cached reads heavily discounted (up to ~90%) against a cache-write surcharge and a short default time-to-live (~5 minutes); OpenAI's is automatic, applying a ~50% discount on repeated prefixes with no write penalty. But mind the last misconception: caching cuts prefill cost on repeated prefixes; it does nothing for decode, and the KV cache still has to physically live in GPU memory. Caching makes long prefixes cheaper — not long context free.

Bookending the Interface section

We closed Substrate with a frozen function that can't see the world. Interface is the answer to "so how do you talk to it?" — and the answer always routes through this one door. The context window is the only write surface a frozen function has; it's bounded for real mechanical reasons; and its usable size is smaller than its advertised size.

Everything else in this section is a refinement of how you fill that window. Roles (next post — system/user/assistant) are about how the window is structured. Sampling and temperature are about how tokens come out. Streaming is about when you see them. Structured output is about constraining what comes out. And further on, RAG is about what goes in, and tools are about expanding what can go in. Different verbs, same object: the context window. Learn to budget it, and you've learned the core of working with language models.

Further reading

Primary papers

• Liu, Lin, Hewitt, Paranjape, Bevilacqua, Petroni, Liang. Lost in the Middle: How Language Models Use Long Contexts — arXiv:2307.03172 (TACL 2024). The U-shaped curve; a 20-point-plus swing for GPT-3.5-Turbo; middle-of-context can fall below the 56.1% closed-book baseline.
• Hsieh, Sun, Kriman, Acharya, Rekesh, Jia, Zhang, Ginsburg (NVIDIA). RULER: What's the Real Context Size of Your Long-Context Language Models? — arXiv:2404.06654 (COLM 2024). Effective vs. claimed length; the 85.6% threshold; GPT-4-1106 and Llama 3.1 70B effective at ~64K despite 128K claims.
• Su, Lu, Pan, Murtadha, Wen, Liu. RoFormer: Enhanced Transformer with Rotary Position Embedding (RoPE) — arXiv:2104.09864 (2021).
• Press, Smith, Lewis. Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation (ALiBi) — arXiv:2108.12409 (2021).
• Chen, Wong, Chen, Tian. Extending Context Window of Large Language Models via Position Interpolation — arXiv:2306.15595 (2023).
• Peng, Quesnelle, Fan, Shippole. YaRN: Efficient Context Window Extension of Large Language Models — arXiv:2309.00071 (2023). See also EleutherAI's explainer, Extending the RoPE.

Benchmarks & evaluations

• Kamradt, G. Needle In A Haystack — Pressure Testing LLMs. The Dolores Park needle; Claude 2.1 / GPT-4 Turbo degradation depths.
• Hong, Troynikov, Huber (Chroma Research). Context Rot: How Increasing Input Tokens Impacts LLM Performance (July 2025). 18 models; non-uniform degradation with input length.

Vendor documentation (figures move — verify against the current model page)

• Anthropic — "Context windows," "Prompt caching," and "Effective context engineering for AI agents" (platform.claude.com/docs and anthropic.com/engineering); "Introducing Claude 2.1" (the 200K announcement).
• OpenAI — "Introducing GPT-4.1 in the API" (1M context); GPT-5 model pages; prompt caching docs.
• Google — Long context; the Gemini 1.5 and 2.5 announcements (blog.google).
• Meta — "The Llama 4 herd" (ai.meta.com/blog) — the 10M Scout window, 256K training length, and iRoPE.

Explainers

• Tobi Lütke (X, June 18, 2025) and Andrej Karpathy (thread, June 25, 2025) on "context engineering."
• LangChain, "The rise of context engineering" (blog.langchain.com) — the write / select / compress / isolate framing.