Notes on agent runtimes and inference.
Serving LLM requests
Twelve questions, from a session’s first message to the cost of serving it. Start with the primer, or pick the question you came with.
Why are prefill and decode different workloads?
Why weight reuse, bandwidth and batching change the hardware needed to serve a request.
Did we actually reuse the previous turn’s cache?
Compare consecutive prefixes, usage denominators and the charges created when output returns as input.
Where did the time and money go in one agent turn?
Define useful traces, token timing and session cost before selecting an open-source observability backend.
Can we keep a cache warm—and is it worth it?
Separate retention policy, matching traffic, warm-up calls and total session economics.
How does DeepSeek share its KV cache?
Follow active state, retained storage and the extra restore path that DualPath adds.
What runs underneath an inference API?
Separate request formats, serving engines, cache allocation, transfer and retained storage.
Why do LLMs generate one token at a time?
How choosing each token creates a dependency, how speculative decoding checks several proposed tokens together, and when that trade saves time.
Why can't we have 10 million tokens of context?
What a context window holds, why longer requests cost more, and how agent software chooses what to keep.
What does a million-token request cost?
Work through explicit token-price and KV-size assumptions without confusing an API bill with infrastructure cost.
What should an agent keep when its context fills?
Separate durable history, active input, compaction checkpoints and the prefix that can be reused.
Why might cheaper AI lead to more AI work?
A working mental model connecting unit cost, useful tasks, measurement and changing bottlenecks.
Runtime experiments
a retry needs a name
A small agent API experiment in giving a repeated submission a stable identity.


a tool needs a room
Running one useful JavaScript tool inside a bounded local gVisor sandbox.


what survives a restart
A process can disappear while a completed session remains usable. The boundary between saved work and unfinished execution matters.


what the model can see
A saved conversation and the next model request are two different things. A small local experiment makes the boundary visible.


Archive
Earlier posts, kept for reference.