A prompt and an answer pass through the same model. Why would a serving system put their work in different pools?
The difference starts with how many positions are available together. A long prompt supplies many known tokens. Ordinary decode adds one position per active sequence at a time. That changes how much useful arithmetic shares each memory read.
If the terminology is new, the introductory article follows a message through tokenization, attention and the KV cache. Here, I want to keep one model operation fixed and change the workload around it.
What does a weight read buy us?
Consider a linear projection: multiply token representations by a learned weight matrix. The GPU moves tiles of that matrix into faster local storage and uses them for arithmetic.
During prefill, a tile can contribute to many prompt positions. During a small decode batch, fewer rows share it. Loading weights is still work even when only a small amount of new text is produced.
One useful quantity is arithmetic intensity: operations divided by bytes moved. A machine also has a ratio between arithmetic throughput and memory bandwidth. Below that ratio, moving data can limit an operation; above it, arithmetic can become the limit. This is a simplified way to reason about a kernel, not a complete serving-time model. NVIDIA’s matrix-multiplication guide
How much work shares a weight read?
Assume a 1,024 × 1,024 weight matrix with two-byte values. Count one weight-matrix read plus input/output bytes; ignore cache hierarchy and other model operations.
| Operation | Rows | Arithmetic intensity |
|---|---|---|
| Prefill projection | 1,024 | 341.3 FLOPs/byte |
| Decode projection | 1 | 1 FLOPs/byte |
The example counts a weight-matrix read and input/output bytes for one projection. It leaves out on-chip cache reuse, attention, collective communication and kernel overhead. Its numbers are calculations under those assumptions, not a benchmark of a named GPU.
Decode still has computation
Decode computes new representations, queries, keys and values. Its attention also reads the prior KV state. The useful shorthand is therefore conditional: long uncached prefill often has enough parallel arithmetic to be compute-bound, while low-batch decode often runs into memory bandwidth.
A larger decode batch can reuse weight reads across requests. A longer context increases attention’s KV traffic. A mostly cached prompt may require little new arithmetic but a substantial restore from another memory tier. There is no single bottleneck label that applies to every model, batch and context length.
This is why “memory-heavy” needs another question. Is there insufficient capacity to hold the state, insufficient bandwidth to read it quickly, or a transfer between workers on the critical path? Those imply different changes.
Why separate the pools?
A large prefill can delay the next small decode step when they compete for the same execution resources. Separate pools allow different batching, parallelism, scheduling and capacity allocation. The goal might be a shorter wait for the first token, more regular subsequent tokens, or greater throughput while meeting both constraints.
Separation also adds work. Compatible KV has to reach the decode worker. The model weights needed for execution may be replicated across the pools. Networking, routing and queueing now influence the result. DistServe
Different pools need not use different GPU models. DeepSeek’s February 2025 V3/R1 account described H800s in both: smaller prefill deployment groups and wider decode groups with different expert parallelism. That is a dated implementation example, not a description of every present request. DeepSeek’s overview
What is Cerebras betting on?
Cerebras takes a different approach to keeping data close to computation. Its October 2026 deployment account identifies 44 GB of on-chip SRAM on each WSE-3T wafer. Large models are distributed across multiple wafers; the article’s illustrative mapping budgets separately for weights and state. The 44 GB figure is not the total memory of an arbitrary full serving system. Cerebras’ engineering account
My reading of that design is that very fast local memory is valuable when repeated reads would otherwise dominate. It does not remove finite capacity, communication, long-prompt arithmetic or the need to evaluate answer quality. Its published performance claims also have their own workloads; they are not measurements from this project.
What would I compare?
I would hold the model, precision, prompt/output lengths and latency target fixed. Then vary batch/concurrency and record first-token latency, inter-token behavior, completed requests, peak state and cost. Disaggregated runs also need transfer bytes and queueing time.
A GPU-utilization percentage alone would not answer whether users received useful tokens sooner. The question is which resource limits the actual workload and whether the proposed change improves the service objective.