A prefix cache hit can save arithmetic and still leave a lot of work to do. The saved state must be available to the worker that will continue the request.
This is where “the cache” stops being a sufficient architecture diagram. We need to distinguish state in GPU memory, state retained elsewhere and the paths used to move it.
DeepSeek has published several useful accounts of those pieces. They come from different dates and systems, so I want to draw the mechanisms separately rather than pretend they form one exhaustive map of every current API request.
Start with one answer
Prefill processes the supplied positions and produces state used by decode. If the phases run in different pools, compatible state has to cross that boundary before the decode worker can continue.
The decode worker’s active KV belongs near the computation using it. Shared backing storage serves another purpose: retaining state that may be useful to later requests. A distributed retained cache does not imply that every attention step reads its whole history from remote SSDs.
Follow the state between serving pools
The figure’s first path is the normal handoff. The second adds restoration for a later matching request. The third highlights the additional route described by DualPath.
What does 3FS provide?
DeepSeek’s 2024 caching announcement described disk-backed reuse. 3FS is a distributed filesystem built around SSD storage and high-speed networking, including RDMA. It can provide shared capacity and aggregate storage throughput.
A filesystem is not the entire inference cache service. The serving system still needs to identify compatible prefixes, index blocks, choose where to execute, coordinate transfer and decide what to retain or evict. A directory full of tensors would not answer those scheduling questions on its own.
What does the newer hierarchy describe?
The pinned DeepInfer deployment Q&A describes active GPU KV, a separate host-memory path for sliding-window state and reusable paged block state backed by 3FS. It explains asynchronous restoration and layer-wise transfer overlapped with computation.
Those distinctions matter. Active state, a sliding-window representation and reusable retained blocks need not have the same layout or transfer policy. Compatibility includes the model, positions, numerical representation and configuration. A cache hit is useful only if the continuing computation can consume what was restored.
The public DeepInfer release excludes infrastructure-coupled production pieces. Its disclosure is informative without making the entire production cache and load balancer reproducible from the released engine alone.
Why would data go through decode toward prefill?
In DualPath, the studied high-reuse agent workload saturates prefill workers’ storage-network interfaces while decode-side storage bandwidth is underused.
The additional route loads retained state through decode hosts, then forwards it toward prefill over the compute network using RDMA. That can draw on bandwidth which would otherwise be idle. Layer-wise transfer, scheduling and traffic isolation are part of the proposal.
This direction can look backwards if we only remember the ordinary prefill-to-decode handoff. It is a different transfer job: restore a retained prefix into the path that will extend it. Once usable state is ready, ordinary continuation toward decode still applies.
The paper addresses a specific bottleneck under its workload. It does not establish that every public DeepSeek request follows that route or that transfer is free.
What changes after a hit?
Imagine a mostly uncached prompt. A prefill worker has substantial arithmetic to perform. Now imagine the same long prefix is reusable. Its arithmetic falls, but reading and restoring the saved state remains. The bottleneck can move from the GPU’s compute units to storage interfaces, host memory or network links.
That is a success exposing the next constraint. Measuring only cached-token percentage would not tell us whether restore time now dominates. We would also want restored bytes, paths, queueing, transfer overlap and end-to-end latency under the actual traffic pattern.
Does this mean a worldwide cache?
No such conclusion follows from shared storage. A disclosure about adjacent data centers does not establish reuse across arbitrary geographic regions, customers or API products. Retention, permissions, namespaces and routing remain deployment contracts.
It is also worth separating names: Mooncake comes from Moonshot/Kimi. Projects can integrate one another’s transport and storage components, but that does not make them interchangeable organizations or architectures.
The lesson for the personal runtime is a boundary to observe, not a reason to reproduce a fleet immediately. Saving a request in PostgreSQL and retaining its KV in an inference service are separate systems. Before adding distributed state, establish exactly which prefix was sent, what usage was reported and which location or timing constraint is being investigated.