Suppose a model request reports that about 91% of its input was read from cache. Is that good? Did we lose some of the previous turn’s state?
The percentage alone cannot answer that. We first need to know what was counted.
An agent usually sends earlier material again along with something new: a user follow-up, a tool result, another instruction or a retrieved file. The new material changes the size of the current request even if the entire old prefix remains reusable.
Two questions that sound like one
Take a synthetic example. A previous comparable prefix contains 1,000 tokens. The next request preserves it and appends 100 new tokens. Assume the provider reports 1,000 cache-read tokens.
The read fraction of the current input is 1,000/1,100, or 90.9%. Coverage of the previous candidate prefix is 1,000/1,000, or 100%.
Those are two valid descriptions of the same example. One measures the composition of the current request; the other asks how much prior material was covered by the reported read count. Neither, by itself, exposes the worker’s physical memory state.
Two denominators, one cached prefix
Previously sent: 1,000 candidate-prefix tokens. Assume every matching token is reported as a read.
| Metric | Calculation | Result |
|---|---|---|
| Current-input read fraction | 1000/1100 | 90.9% |
| Previous candidate coverage | 1000/1000 | 100% |
| Matching-prefix coverage | 1000/1000 | 100% |
The example deliberately assumes that reported reads correspond to all matching positions. Actual providers can impose cacheable-length thresholds, block alignment, lookup boundaries, isolation and routing behavior. A missing denominator is unavailable, not automatically 100%.
The request is the evidence
Comparing the chat messages we can see is not enough. The provider also receives instructions, tool definitions, formatting and settings. A date placed near the beginning, reordered tool schemas or a changed system instruction can change the prefix before the interesting conversation begins.
For each adjacent request pair I want to save the exact outgoing payload, model/configuration, relevant cache controls and raw usage. With an appropriate tokenizer or provider token-counting path, we can identify the longest matching prefix. A byte comparison is useful for finding a serialization change but is not automatically a token count.
Then distinguish three quantities: the prior candidate prefix, the compatible matching prefix, and reported cache reads. Explain which of them each ratio uses. If an API returns only aggregate usage, do not pretend it identifies the origin or physical residency of every block.
OpenAI’s cache guide documents model-dependent matching, routing and usage behavior. Anthropic’s usage guide separates cache creation, reads and ordinary input. The accounting contracts need to be checked for the chosen endpoint.
Generated output becomes later input
There is another cost that is easy to overlook. An assistant message containing a tool call is output from one request. After the tool runs, that message returns as part of the next input.
Hunter Paulson’s cache-write billing article follows this span across the boundary. It can incur an output charge when generated and a cache-write input charge on its next appearance. His proposal is to retain that generated state so the later use can be a read instead.
That is not two output-rate charges. Billing categories have different rates, and the contract can change. The correct conclusion for a real session comes from its actual usage and the applicable price schedule.
For an illustrative schedule of 10 units/M output, 2.5 units/M cache writes and 0.1 units/M reads, 1,000 tokens cost 0.010 units to generate. A later write adds 0.0025; a read would add 0.0001. This isolates one span and excludes other prompt tokens, tools, retries and future calls. It is arithmetic, not an invoice or an internal cost estimate.
What does our event log help with?
The personal runtime saves durable events and exact request snapshots separately. That lets us establish which history was selected for a particular call. It currently uses an offline fixture in its public demonstrations, so those receipts do not establish a provider cache hit.
The next controlled experiment needs real personal-provider usage for a fixed synthetic prefix. Compare an immediate continuation, an early instruction change, a tool-definition change and a compaction. Keep model/settings fixed, preserve raw responses and include unsuccessful attempts in the accounting.
I would report current-input read fraction and prior-prefix coverage alongside total charges. A dashboard showing only requests with any cache hit would miss partial-prefix losses. A dashboard showing only cached/input can also make a healthy growing session look worse as new material is added.
The useful outcome is not a larger percentage in isolation. It is being able to explain which work was eligible for reuse, what the API reported, and what the whole session cost.