Why can't we have 10 million tokens of context?

A token is a unit of text that a model processes. It might be a word, part of a word, or punctuation. Code is split into tokens too. context is the material the model can use for one request: instructions, conversation, file contents, and tool results supplied to it.

The context window is that request's token budget. Input uses part of it; generating an answer needs room too. When a model counts internal reasoning as output, those tokens share the output allowance with the visible reply. The application needs to leave room for both. The exact input, output, and total limits depend on the model and API.

A saved chat transcript is a different thing. An application can store a conversation longer than one window, then select which parts to send next. A turn can still exist in the saved record while being absent from the model's current request.

The software around the model chooses that input, runs tools, and saves results. It is often called an agent harness, or runtime. The model handles a request; the harness carries the session forward.

If an agent misses an earlier decision after a long session, a larger window seems like the obvious fix. Keep the entire conversation, every file it read, every tool result, and every failed approach. Give the model ten million tokens and let it work.

That would let more material fit. It leaves three questions: what does processing it cost, can the model use the right evidence, and what happens when the saved session grows beyond that window too?

Can a model already accept ten million tokens?

Google reported successful research tests up to ten million tokens when it announced Gemini 1.5 in February 2024. The announcement distinguished those experiments from its one-million-token production capability and limited developer preview. the original announcement

So the interesting question is why ten million tokens are not a cheap, reliable default for every agent request.

A maximum accepted length says something about capacity. It does not establish uniform task accuracy, response time, or the number of simultaneous requests a service can afford to keep active.

Where does the 255k number come from?

I started with the question “why does OpenAI stop around 255k instead of serving a million?” the public definitions do not support that as a general premise.

As checked on November 11, 2026:

These are the separate limits in the GPT-5.3-Codex and Sol cards. Sol's card also states a 500k default window in Codex and ChatGPT Work. A million-token total window does not mean a million input tokens plus an unrestricted reply.

The client adds another budget. It can shorten the next input before reaching the limit, an operation called compaction. A public Codex regression test distinguishes a configured 272,000-token window, 258,400 usable tokens at 95%, and a normal automatic-compaction threshold of 244,800 at 90%. The reserve leaves room for overhead and output. Those values describe that source example, not a measurement of this session.

There is even a snapshot discrepancy worth preserving: that revision's bundled Sol catalog still starts at 272k, while the model card says 500k for Codex. The client can load a provider catalog and apply bounded overrides. A bundled default alone cannot establish the active product setting.

I did not find an authoritative exact 255,000-token limit or a public explanation for choosing that number. Nearby client budgets explain why a quarter-million figure can appear; they do not establish a universal quality cliff or OpenAI's internal rationale.

Why does a longer request need more hardware?

A model first processes the supplied input, a stage called prefill. The last prompt position can predict the first output token. To continue, the model processes that new token and predicts the following one. Repeating that step is called decode.

To see why length matters, we need one part of the model: attention. At an attention layer, the model turns token representations into queries, keys, and values. A query is compared with keys; the resulting scores determine how to combine the values. This is how a position can use information from other positions. the preceding post follows that calculation in more detail.

When generating the next token, the service can reuse keys and values already computed for earlier positions. That retained state is the key-value cache, usually shortened to KV cache. It saves recomputation, but it occupies hardware memory. It is computed model state, separate from the text an application saves in its transcript.

A longer prompt therefore asks for more attention calculations and more retained state.

For full causal attention over a prompt of length n, each position can attend to itself and preceding positions. There are approximately n² / 2 query–key pairs per head. Going from one million to ten million positions creates roughly one hundred times as many pairs during prefill.

Other work scales differently. The transformations applied separately at each position, including projections and feed-forward layers, process ten times as many positions. A conventional KV cache grows about tenfold. During decode, each new query has a ten-times-longer history to attend over. These are separate terms, not a prediction that the entire request becomes exactly one hundred times slower or more expensive. the inference scaling chapter

FlashAttention is an implementation of attention that reduces expensive memory traffic and avoids storing the entire attention-score matrix in accelerator memory. Exact dense attention still performs the quadratic attention computation. Its storage improvement does not make that arithmetic linear. FlashAttention

We can size the retained keys and values without running a model. The inputs are the number of positions, layers, stored key/value heads, numbers per head, and bytes per number. A KV head produces one key vector and one value vector per position in each attention layer.

Reuse the preceding post's hypothetical layout: 32 layers, eight KV heads, head dimension 128, and two-byte elements for both keys and values. The head dimension means each stored key or value contains 128 numbers.

1KV bytes = positions × layers × KV heads × head dimension2         × 2 (keys and values) × bytes per element

The GB and GiB columns below express the same bytes. GB uses powers of 1,000; GiB uses powers of 1,024.

This is one sequence, before weights, temporary buffers, allocator overhead, or additional requests. It is explicit sizing arithmetic, not the shape of an OpenAI model or a memory allocation we measured. The runnable calculator uses the same assumptions.

Different model designs change what needs to be stored or calculated. Grouped-query attention lets multiple query heads share stored keys and values, reducing the KV-head count. DeepSeek-V2's multi-head latent attention compresses the cached state. Sparse attention visits fewer positions. Linear attention can carry a recurrent state instead of keeping a growing history of keys and values; hybrid models combine different paths.

These approaches change the memory and computation requirements in different ways. None establishes unlimited, equally accurate memory. DeepSeek-V2's MLA, Linear Transformers

Does fitting more text mean using it well?

The model must choose useful information from that larger history. Many irrelevant passages can resemble the question, repeat old instructions, or contradict a later correction. The answer may require joining several facts far apart rather than finding one matching sentence.

This makes the task harder even when every token fits. Attention is learned retrieval and combination, not a guarantee that each relevant fact will receive the right weight. Longer input also tests the length and position patterns the model learned during training. There is no requirement that relevance receive 1/n of the attention, and no universal token count at which reasoning suddenly fails.

The evaluation history helps separate the problems:

• Lost in the Middle found position-dependent performance in the models and retrieval/QA tasks it tested. Information near the beginning or end was often easier to use than information in the middle.
• RULER added tracing, aggregation, and multiple-needle tasks. Strong performance on a simple planted needle did not establish equally strong performance on those harder tasks as context grew.
• NoLiMa removed easy literal matches between questions and supporting text. Retrieving evidence through less obvious associations became much harder for its evaluated models at longer lengths.
• LongPiBench supplies useful counterevidence to a universal story: most tested models resisted the classic single-piece middle effect, while distances between multiple relevant pieces still introduced substantial biases.

These studies test particular model snapshots and workloads. They do not prove that every current model degrades smoothly with every additional token. Useful evidence can improve an answer; unnecessary context can obscure it. A one-fact retrieval score cannot stand in for a week-long coding task with corrections, changing files, and unresolved decisions.

How do cost and compaction fit into this?

The service needs accelerator time, state capacity and data movement. Its bill and its infrastructure cost are different quantities. The million-token cost article works through the arithmetic with explicit assumptions. The hardware article examines bandwidth and the Cerebras memory tradeoff.

A session can also outlive one model window because the harness constructs a smaller next input from a larger saved record. Selecting turns, summarizing older material and retrieving evidence are different policies. The compaction article compares public Codex, OpenCode and Pi mechanisms and explains why a nearby client budget does not establish a universal 255k model limit.

What could we test in the personal runtime?

Our personal runtime experiment already separates saved events from selected model context. It currently selects complete turns; summary generation and retrieval are proposed additions. That gives the experiment a clear baseline.

The deterministic offline fixture already shows this boundary: omit the turn containing a codeword and the response reports it missing; select that saved turn again and it returns. The existing context-selection check passed again for this post; the saved request snapshots expose the same selection boundary. It demonstrates request selection, not forgetting by a real model.

I would build synthetic sessions with durable constraints, later corrections, tool failures, and evidence the final task must combine. Compare recent turns only, a checkpoint plus recent turns, and retrieval from the original record under equal active-context budgets. Use the growing full transcript as a reference only while it fits. Move the necessary facts around and add irrelevant material without changing the required answer.

Measure task completion and constraint violations alongside input/output tokens, compaction calls, cache behavior, first-token latency, and total charges. Count facts the summary lost and facts retrieval recovered. Hold the model, task, and sampling setup fixed.

That experiment has not been run. The checks accompanying this post verify sizing, source-budget arithmetic, and the hypothetical bill; they measure no model's long-context accuracy.

A larger window is useful capacity. A memory policy decides which evidence deserves that capacity on the next request, and how the agent gets the rest back when it needs it.