Splitting the LLM in Two: Prefill/Decode Disaggregation for SLO-Grade Serving
Splitting the LLM in Two: Prefill/Decode Disaggregation for SLO-Grade Serving A single serving instance that mixes prompt processing with token generation is simple, and for many deployments it is the right call. But once you must meet latency targets at a sustained request rate, the two phases of an LLM request start fighting each other on the same GPUs. Prefill/decode disaggregation resolves that fight by splitting the phases onto separate pools of machines, connected by a fast KV-cache handoff. This article explains what the disaggregation actually buys — and what it does not — based on the primary systems literature (DistServe, Splitwise, Mooncake) and the current official documentation of production engines (vLLM, NVIDIA Dynamo).