Prefill and decode in LLM inference

Originally written in Japanese. This is a translation of the same piece.

Introduction

At KubeCon + CloudNativeCon Japan 2026 there were several sessions on improving LLM inference performance. A distinction these sessions seemed to treat as shared background was that inference has two phases: prefill and decode. Even though it is the same model running on the same GPU, in the first half compute speed is said to be the limit, while in the second half it is the memory reads that cannot keep up. Right after hearing it, this idea of the bottleneck swapping mid-process did not quite click for me.

This article is a record of digging into it until I was satisfied I understood.

Generation proceeds one token at a time

An LLM does not emit its answer in one piece; it normally generates tokens (units roughly corresponding to words or fragments of characters) one at a time. It looks at the input and the tokens it has produced so far, picks the next one, appends it to the end, and picks the next again, over and over. A chat response appearing gradually from the beginning is not a loading animation; many systems stream the generated text as it is produced, in generation order.

Done naively, this loop would redo the computation for all past tokens on every step. So in practice, inference stores the intermediate results computed for each token in GPU memory and reuses them when picking the next token. This store of intermediate results is the KV cache¹. Prefill and decode can be restated as the phase that “builds the KV cache” and the phase that “uses it while extending it one entry at a time.” Note that the explanation from here on assumes the commonly used setup: a dense decoder-only Transformer that attends to all past tokens, run with this ordinary KV-cache decoding².

Prefill reads the input in parallel

Prefill is the phase that processes the user’s entire input (the prompt), producing the KV cache and the first token. Since all of the input tokens are available from the start, there is no need to handle them one by one; the computation for every token can be issued in parallel.

So why would a phase that merely reads the input hit a ceiling on compute speed? A GPU is hardware that performs best when large matrix computations are fed to it in bulk. Prefill fits this sweet spot, and because the same weights apply to every token, weights read once can be reused for the computation of hundreds to thousands of prompt tokens. The longer the input, the more computation is done per unit of weights read. Once that ratio grows large enough, the ceiling moves from memory reads to compute speed. A state where compute speed itself is the ceiling is called compute-bound. For a typical prefill over a prompt of some length, the GPU’s compute units stay busy almost without pause. From the user’s point of view, the length of prefill shows up as the model-execution part of the wait between hitting send and the first output token arriving³.

Decode writes the output one token at a time

Decode is the phase that generates the second token onward, one at a time. The next token’s computation cannot start until the previous token is decided, so there is no processing things in bulk the way prefill does.

What matters here is how little computation one token involves. Generating a single token requires reading the model’s entire weights (parameters) and the KV cache out of GPU memory. Yet the computation performed with those freshly read weights amounts to just one token’s worth. Take a dense 7-billion-parameter model (about 14 GB of weights at 16-bit precision) serving a single request at a time (batch size 1), and estimate with the H100 SXM’s catalog numbers.

Reading the weights: about 14 GB ÷ roughly 3.35 TB per second, which is about 4 milliseconds
Computing with them: roughly 14 billion floating-point operations (about twice the parameter count) ÷ FP16 throughput of about 1,000 trillion per second (the figure without the sparsity doubling), which is a bit over ten microseconds

A gap of roughly 300×. Both are ideal-condition lower bounds, not measurements. As the context grows, KV-cache reads add to the reading side and attention computation to the computing side. Even so, the estimates are enough to gauge that reading the weights outweighs computing with them by more than two orders of magnitude. While few requests are being served at once, the compute units sit idle most of the time, waiting on reads from memory. This state is memory-bound.

Hearing “memory is the bottleneck” may bring capacity shortages to mind, but what is described here is a different constraint. Assuming the weights and the KV caches of the requests in flight fit in GPU memory, the ceiling is still the speed (bandwidth) at which what is already there gets read out for every single token. Capacity is not irrelevant either: as contexts grow longer or concurrent requests multiply, the KV cache swells, and it surfaces as a separate constraint on how many requests fit at once. Decode speed sets how fast output tokens are generated, which users experience as how fast the characters of the response flow.

The two phases side by side

	Prefill	Decode
Role	Read the input, build the KV cache	Write the output one token at a time
Parallelism	Input tokens computed in bulk	Sequential, waiting on the previous token
Typical bottleneck	Compute speed (compute-bound)	Memory reads (memory-bound)
Where users feel it	Wait to the first output token	Speed at which characters flow

That said, being compute-bound or memory-bound is not a property glued to a phase’s name. It is decided by how much computation is done per unit of data read, and the boundary moves with the prompt length, the number of concurrent requests, and the model’s shape. What the table shows is the typical case, where that ratio swings to an extreme.

Why this distinction comes up so often

Characteristics being almost opposite also means an optimization that helps one phase does not help the other in the same way. For example, processing multiple users’ requests together (batching) lets decode reuse weights read once across the requests, giving work to compute units that were otherwise waiting on reads. What this grows, though, is the total number of tokens the system generates per second (throughput); the speed of the text as seen by any single user does not necessarily improve. Prefill, on the other hand, can saturate the compute units with a single long prompt, so the headroom from batching is comparatively small.

Taking this difference further leads to prefill/decode disaggregation (disaggregated serving), where prefill and decode run on separate GPU pools. When the two kinds of work share a GPU, in a naive setup, every time a long prompt’s prefill cuts in, other users’ decode stalls and the flowing text freezes. There is a way to stay co-located. Chunked prefill chops prefill into small pieces and mixes them in with decode. Separating them instead removes the interference itself and lets the GPU type and count be chosen independently for each phase. In exchange, a new cost appears: the KV cache built on the prefill side has to be shipped to the decode-side GPUs. Which wins depends on the traffic and the model, and also on the effective bandwidth and placement between the GPU pools. The reason a Kubernetes conference keeps coming back to this distinction is, I suspect, that deciding how to split and how to scale is precisely the infrastructure side’s job.

Closing

The opening puzzle, “the same inference, yet the bottleneck swaps midway,” can be answered like this: under typical conditions, the amount of computation per unit of data read differs to an extreme between the phase that reads the input in parallel and the phase that writes the output one token at a time. With this distinction in mind, rereading the session material let me file each technique — KV cache management, batching, prefill/decode disaggregation — under “which phase, and which resource, it rescues.”

References

The name comes from storing the two kinds of values, Key and Value, used by the Transformer’s attention mechanism. Without going deeper, the picture of “a savings account of past computation” is enough to follow the rest. ↩
“Dense” means every token’s computation uses the full set of parameters. In MoE (Mixture of Experts), which activates only a subset, “read the entire weights” does not hold as stated; in speculative decoding, where a small draft model proposes a candidate sequence and the main model verifies it in parallel and can accept several tokens at once, “always one token at a time” does not. Configurations such as sliding-window attention, which restricts attention to a recent window, also fall outside the assumption of attending to all past tokens. ↩
This “time until the first output token arrives” is what the metric TTFT (Time to First Token) measures. Besides prefill, TTFT also includes time spent queueing and in transit, and under heavy load those parts can dominate. Before the first character shows on screen, client-side rendering adds a little more. ↩