Notes on agent runtimes and inference.

Serving LLM requests

Twelve questions, from a session’s first message to the cost of serving it. Start with the primer, or pick the question you came with.

  1. Why are prefill and decode different workloads?

    Why weight reuse, bandwidth and batching change the hardware needed to serve a request.

  2. Did we actually reuse the previous turn’s cache?

    Compare consecutive prefixes, usage denominators and the charges created when output returns as input.

  3. Where did the time and money go in one agent turn?

    Define useful traces, token timing and session cost before selecting an open-source observability backend.

  4. Can we keep a cache warm—and is it worth it?

    Separate retention policy, matching traffic, warm-up calls and total session economics.

  5. How does DeepSeek share its KV cache?

    Follow active state, retained storage and the extra restore path that DualPath adds.

  6. What runs underneath an inference API?

    Separate request formats, serving engines, cache allocation, transfer and retained storage.

  7. Why do LLMs generate one token at a time?

    How choosing each token creates a dependency, how speculative decoding checks several proposed tokens together, and when that trade saves time.

  8. Why can't we have 10 million tokens of context?

    What a context window holds, why longer requests cost more, and how agent software chooses what to keep.

  9. What does a million-token request cost?

    Work through explicit token-price and KV-size assumptions without confusing an API bill with infrastructure cost.

  10. What should an agent keep when its context fills?

    Separate durable history, active input, compaction checkpoints and the prefix that can be reused.

  11. Why might cheaper AI lead to more AI work?

    A working mental model connecting unit cost, useful tasks, measurement and changing bottlenecks.

Runtime experiments

Archive

Earlier posts, kept for reference.