Splitting the LLM in Two: Prefill/Decode Disaggregation for SLO-Grade Serving

A single serving instance that mixes prompt processing with token generation is simple, and for many deployments it is the right call. But once you must meet latency targets at a sustained request rate, the two phases of an LLM request start fighting each other on the same GPUs. Prefill/decode disaggregation resolves that fight by splitting the phases onto separate pools of machines, connected by a fast KV-cache handoff. This article explains what the disaggregation actually buys — and what it does not — based on the primary systems literature (DistServe, Splitwise, Mooncake) and the current official documentation of production engines (vLLM, NVIDIA Dynamo).

All performance figures below are cited from the named papers and official docs together with their testbeds; none were reproduced locally. The article closes with a deterministic trace-replay experiment design that lets you measure the trade-off on a single machine.

Two latencies, one queue#

An LLM request has two phases with opposite resource profiles:

• Prefill processes the whole prompt in one forward pass to produce the first token. It is compute-bound: for a 13B model, a 512-token prompt already makes an A100 near compute-bound [1].
• Decoding generates subsequent tokens one step at a time. Each step reads the whole KV cache, so it is memory-bandwidth-bound and compute-idle [1, 2].

Two latency metrics follow directly. Time to first token (TTFT) is dominated by the prefill phase; time per output token (TPOT) is the average step time of decoding. The vLLM documentation calls TPOT inter-token latency (ITL) and treats TTFT and ITL as the two independently tunable serving targets [4].

The problem: in a colocated (monolithic) instance, both phases share one GPU and one queue. Continuous batching deliberately mixes them, so a long prefill step stretches the decode steps batched with it (TPOT blows up), and decode work in the batch delays prefill (TTFT blows up). DistServe’s measurements on a 13B model show the effect sharply: at 90% SLO attainment on one A100, a monolithic system sustains roughly 1.6 requests/s, while the same GPU serving only prefill sustains 5.6 requests/s and only decode 10 requests/s. Two prefill GPUs plus one decode GPU would serve 10 rps — 3.3 rps per GPU, 2.1× the monolithic number [1].

Chunked prefill is the classic mitigation: split long prefill into chunks and interleave decode steps. It reduces but does not eliminate interference, it trades TTFT for TPOT, and it re-reads the KV cache of all previous chunks for every subsequent chunk — splitting one prefill into $N$ chunks costs $O(N^2)$ KV cache loads instead of $O(N)$ [1]. The vLLM docs make the same point from the production side: with chunked prefill “it’s hard to figure out the correct chunk size value”, so disaggregation is “a much more reliable way to control tail ITL” [4].

This is where goodput enters. Raw throughput is requests or tokens per second regardless of latency. Goodput is the maximum request rate served while meeting latency SLOs — DistServe defines per-GPU goodput as the max rate sustaining an SLO attainment goal such as 90% [1]. Maximizing goodput, not throughput, is the objective of every disaggregated system in this article.

Figure 1 — Throughput vs goodput (illustrative; pattern per DistServe Fig. 1 [1])

  served (req/s)
  10 ┤ ─────────────────────────╱  raw throughput (saturates, ignores latency)
   9 ┤                         ╱
   8 ┤                        ╱
   7 ┤                       ╱╲  goodput (meets TTFT and TPOT SLOs)
   6 ┤                      ╱  ╲
   5 ┤                     ╱    ╲
   4 ┤                    ╱      ╲
   3 ┤                   ╱        ╲
   2 ┤                  ╱          ╲
   1 ┤                 ╱            ╲
   0 └────────────────┴───────┴──────┴──────► offered load (req/s)
       0              5      7      10
                       └── knee: beyond this, queueing pushes SLO
                           attainment below target; goodput falls

Anatomy of a disaggregated deployment#

Disaggregation keeps a full copy of the model weights in each pool, so prefill and decode instances can use different parallelism (e.g. tensor parallelism for prefill to cut TTFT, pipeline/replication for decode to scale rate) and different GPU types entirely [1, 2]. A router (or cluster-level scheduler) assigns each request to a prefill instance; the prefill instance runs the prompt, then hands the resulting KV cache to a decode instance, which generates the rest of the tokens [1, 2, 3].

Figure 2 — Monolithic vs disaggregated topology (original diagram)

(a) Monolithic: one pool, mixed batches

   clients ──► router ──► [ GPU pool: prefill + decode batched together ]
                              │  a long prefill delays decode steps;
                              ▼  decode work inflates TTFT
                         tokens out

(b) Disaggregated: two pools, one KV handoff

   clients ──► router ──► [ prefill pool ]   compute-bound, small batches,
                              │              TP-heavy parallelism
                              │  KV cache per layer, transferred as
                              │  it is produced (overlapped)
                              ▼
                     ┌──────────────────────────┐
                     │  KV transfer / connector │  NVLink, InfiniBand,
                     │  (e.g. vLLM Connector,   │  or RoCE; optional
                     │   Mooncake Transfer Eng.)│  DRAM/SSD KV-store tier
                     └──────────────────────────┘
                              │
                              ▼
                        [ decode pool ]   memory-bound, large batches,
                              │           older or power-capped GPUs OK
                              ▼
                         tokens out

The handoff is the entire cost of disaggregation: no KV cache needs to move in a monolithic system. Everything else — scheduling, placement, pool sizing — exists to make that handoff cheap or invisible.

The KV handoff: how much, how fast#

How much. The KV cache for one token is:

KV bytes per token = 2 (K and V) × layers × KV-heads × head-dim × bytes-per-element

with GQA models using fewer KV heads than query heads, and FP16 giving 2 bytes per element. Applying this to current model configs (Table 1), a 2k-token prompt of Llama-3.1-8B produces about 256 MiB of KV cache; the same formula gives LLaMA3-70B ≈ 320 KiB per token, consistent with Mooncake’s documentation that a 128k-token LLaMA3-70B KV cache is ≈ 40 GB [5].

Table 1 — KV cache per token, computed with the formula above from the models’ public configs; the Llama-3-70B row matches Mooncake’s documented ≈40 GB for 128k tokens [5].

How fast. The required transfer bandwidth is KV bytes/token × prompt length × arrival rate. DistServe’s worked example: a 512-token request on OPT-66B has ≈ 1.13 GB of KV cache; at 10 requests/s that is 11.3 GB/s ≈ 90 Gbps of sustained transfer just to make the handoff invisible [1]. Their testbed had only 25 Gbps cross-node links (4 nodes × 8×A100-80GB), so placement had to colocate prefill-decode pairs on NVLink-connected GPUs where possible [1].

The path. Three designs are representative:

• Serialized transfer: the whole KV cache moves after prefill finishes and before the first decode step. Simple, but the transfer time adds directly to the second-token latency.
• Layer-wise, overlapped transfer (Splitwise): each layer’s KV is shipped with a zero-copy, one-sided RDMA put (implemented with MSCCL++) as the next layer computes. The residual, non-overlapped transfer time is ≈ 8 ms on A100 (200 Gbps InfiniBand) and ≈ 5 ms on H100 (400 Gbps) — under 7% of prompt compute time; for small prompts (< 512 tokens) Splitwise falls back to serialized transfer because the KV is too small to bother [2].
• Multi-NIC transfer engine (Mooncake): the Transfer Engine aggregates several RDMA NICs, picks topology-aware paths, and fails over automatically. It moves 40 GB (a LLaMA3-70B 128k-token KV cache) at 87 GB/s on 4×200 Gbps RoCE and 190 GB/s on 8×400 Gbps RoCE — about 2.4× and 4.6× faster than TCP respectively [3, 5].

Figure 3 — Request timeline and the KV handoff (original diagram)

  arrival      queue         prefill        KV transfer        decode
     │           │              │               │                 │
     ├───────────┼──────────────┼───────────────┼─────────────────┤
     │  router   │  prefill     │  layer-wise   │  first token    │
     │  assigns  │  computes    │  RDMA puts    │  served, then   │
     │  prefill  │  prompt;     │  overlap with │  steady TPOT    │
     │  instance │  KV produced │  next layer   │  (decode pool)  │
     ▼           ▼              ▼               ▼                 ▼
     └───────────┴──────────────┴───────────────┴─────────────────┴──► time

  TTFT  = queue + prefill + residual transfer + first decode step
  TPOT  = time per subsequent decode step (transfer fully overlapped
          by then, or hidden behind batching)

  Measured impact of the residual (Splitwise, coding trace):
    serialized transfer: second-token latency +64%, E2E up to +3%
    layer-wise transfer: second-token latency +16.5%, E2E +0.8%

Scheduling, network, and pool sizing#

Scheduling. The three papers take different routes, all aimed at the same goodput objective:

• DistServe co-optimizes GPU allocation and parallelism for each phase given the TTFT/TPOT pair, then places instances to minimize cross-node KV traffic on the cluster’s actual bandwidth; requests run FCFS per instance, and it uses a simulator over resampled traces to evaluate SLO attainment before committing to a plan [1].
• Splitwise uses two-level scheduling: a cluster-level scheduler (CLS) routes requests with join-the-shortest-queue across prompt/token/mixed pools, while machine-level schedulers (MLS) batch per phase — prompt machines restrict total batch to ~2048 tokens, token machines batch as large as memory allows. Under load, machines flow between pools to avoid fragmentation; at high load everything devolves to mixed batching [2].
• Mooncake is KVCache-centric: the scheduler balances effective throughput against latency SLOs, and under overload it applies prediction-based early rejection rather than letting queues grow unboundedly [3].

Network. The network requirement scales with prompt length × arrival rate (see the 90 Gbps example above). That is why every system here assumes fast fabrics: Splitwise runs over InfiniBand at 200/400 Gbps [2]; Mooncake aggregates 4×200 Gbps or 8×400 Gbps RoCE NICs [3, 5]; DistServe on a 25 Gbps testbed compensated with NVLink-colocated placement [1]. If your cluster only has commodity Ethernet without RDMA, TCP multiplies the transfer time (Mooncake’s 2.4–4.6× gap) and the handoff stops being hidden.

Pool sizing. The right prefill:decode ratio follows the workload’s input/output token distribution, not intuition:

• Splitwise’s iso-power provisioning for the coding trace (large prompts, ~13 output tokens median) is 35 prompt : 5 token machines on H100s; the conversation trace (median ~129 output tokens) wants 25 : 15 [2].
• DistServe’s motivating example needed 2 prefill GPUs per 1 decode GPU for the 13B summary workload [1].
• NVIDIA Dynamo’s docs add a capacity warning: decode-side KV capacity can become the bottleneck, and adding prefill replicas can reduce total system capacity if decode KV is what actually limits concurrency [6].

There is no universal token threshold for splitting — the production documentation says so explicitly, and the pool ratio is a workload property you measure, not a constant [6].

Systems compared#

Table 2 — Prefill/decode disaggregation systems, per their papers and official docs.

The production row deserves emphasis, because the vLLM documentation is explicit about what disaggregated prefilling does not do:

Disaggregated prefill DOES NOT improve throughput.

The value, per the docs, is two-fold: (1) you can tune TTFT and ITL separately — e.g. assign different tensor/pipeline parallelism to the prefill instances without touching decode; (2) you get deterministic tail-ITL control, because prefill jobs no longer get inserted into decode batches. The implementation runs two vLLM instances (prefill + decode) joined by a Connector; the core abstractions are Connector (producer→consumer), LookupBuffer (non-blocking insert, blocking drop_select), and Pipe (FIFO tensor send/recv), with token-id reuse so the decode instance skips re-tokenizing the prompt. Nine connector types are currently documented, from NixlConnector (UCX/GDS backends) and MooncakeConnector to OffloadingConnector and FlexKVConnectorV1 [4]. Related research on chunked prefill with two-level scheduling (TetriInfer, arXiv 2401.11181) reports 38% fewer resources at −97% average TTFT and −47% average job completion time in simulation — a reminder that disaggregation is one point in a larger scheduling design space.

When disaggregation is the wrong tool#

• Small footprints. Disaggregation needs at least two instances (plus a router); on one GPU it is pure overhead. This is a deployment topology, not a library you enable.
• Low concurrency. NVIDIA Dynamo’s docs note that at low concurrency the aggregated configuration wins — there is no interference to eliminate if queues are empty [6].
• Batch workloads without latency SLOs. Splitwise’s own evaluation shows that under sustained high load, disaggregated clusters devolve into the mixed-batching baseline — the goodput advantage is precisely the SLO filter, and batch jobs that don’t need it get nothing [2].
• Weak interconnects. If sustained transfer bandwidth cannot approach KV bytes/token × prompt length × rate (90 Gbps in DistServe’s OPT-66B example [1]), the handoff stops being hidden and you are paying two copies of the weights for a slower system. PCIe-only, single-NIC, or non-RDMA setups put you in this zone.
• Short prompts and short outputs. Splitwise transfers small prompts (< 512 tokens) serially because the overhead is trivial either way — and conversely, the transfer is proportionally most visible when the decode phase is short [2].
• Operational cost. Two pools means two copies of weights, two failure domains, extra observability, and capacity planning that must respect the decode-KV bottleneck [6]. Prefix caching, chunked prefill (with careful chunk-size tuning), and KV stores that serve repeated prefixes can capture part of the benefit with far less machinery.

A deterministic trace-replay experiment#

Cluster numbers (7.4×, 2.35×, +525%) come from specific multi-node testbeds and cannot be reproduced on a workstation. But the mechanism — where the goodput knee sits as a function of pool ratio — can be measured deterministically on one machine, with no wall-clock dependence:

Setup. A small event-driven simulator (~300 lines; either reuse the open-source Splitwise event simulator [8] or write your own) whose only inputs are: (a) a request trace, (b) a fixed profiler table, (c) fixed parameters (link bandwidth, SLOs, pool ratio).

• Traces. (a) ShareGPT, e.g. the sample used by vLLM’s benchmark_serving.py; (b) the Azure LLM inference trace subset released in AzurePublicDataset (arrival time + input/output token counts) [7]; (c) Mooncake’s released FAST’25 trace mooncake_trace.jsonl (arrival times, input/output tokens, remapped block hashes) [5]. The Mooncake trace also enables a prefix-cache extension later.
• Profiler table. Once, on the target GPU (e.g. Llama-3.2-1B FP16 on a single TITAN RTX 24 GB): prefill tokens/s at batch sizes 1/2/4/8 and decode tokens/s at batch sizes 1–64. Record the table and its SHA-256 in the repo.
• Service model. Request lifecycle: arrival → prefill queue → prefill service (batch-aware, from the table) → KV transfer time = KV bytes / link bandwidth (parameter, e.g. 12 GB/s PCIe or 25 Gbps Ethernet) → decode service (from the table). Monolithic baseline: mixed-batch service time from the same profiling.
• Determinism. No wall clock enters the model — every service time comes from the profiler table; events are processed in (arrival_time, request_id) order with fixed tie-breaking; no sampling, no PRNG. Replaying the identical trace and table yields identical output. Store trace + table hashes in the result header.
• Sweeps. Arrival-rate multiplier λ ∈ {0.2 … 2.0}; prefill:decode GPU ratio (e.g. 2:1, 3:1, 1:1, 1:2) at fixed GPU count; SLOs TTFT ∈ {0.5, 1, 2 s}, TPOT ∈ {20, 50, 100 ms}.

Outputs and expected findings. SLO-attainment vs λ curves per configuration, plus raw throughput for reference: at low λ the monolithic curve matches or beats disaggregation (the handoff is pure cost); past the knee, disaggregated configurations hold attainment higher, and the winning pool ratio shifts with the workload’s input/output token distribution — long-input traces want more prefill GPUs, mirroring Splitwise’s 35:5 vs 25:15 provisioning [2]. The experiment’s purpose is to demonstrate the shape (goodput ≠ throughput; pool ratio is workload-dependent), not to produce cluster-equivalent numbers — the per-GPU service rates come from a local profiler, and the simulator should be validated against a two-instance real run (two vLLM instances on one GPU with NixlConnector or LMCacheMPConnector, expected: monolithic wins at low concurrency, tail-ITL control appears under load) before any absolute figure is trusted [4].

Status and dated caveats (2026-08-09)#

• vLLM disaggregated prefill remains experimental and subject to change; the documented connector list and kv_transfer_params API may shift between releases [4].
• All cited numbers are testbed- and era-specific: DistServe and Splitwise were measured on A100/H100-era hardware with 25–400 Gbps fabrics [1, 2]; Mooncake’s figures come from Kimi production and RoCE clusters [3, 5]. They are cited, not reproduced here.
• NVIDIA Dynamo’s documentation is under active development; details were taken from the dev docs and may already have moved [6].
• Mooncake has evolved beyond the FAST’25 paper (Transfer Engine, Store, EP/PG); the paper and repo are the stable references [3, 5].

References#

1. Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, Hao Zhang. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving (OSDI ‘24). arXiv:2401.09670.
2. Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, Ricardo Bianchini. Splitwise: Efficient generative LLM inference using phase splitting (ISCA ‘24). arXiv:2311.18677.
3. Ruoyu Qin, Zheming Li, Weiran He, Mingxing Zhang, Yongwei Wu, Weimin Zheng, Xinran Xu. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving (FAST ‘25). arXiv:2407.00079.
4. vLLM documentation. Disaggregated Prefilling (experimental). https://docs.vllm.ai/en/stable/features/disagg_prefill.html
5. Mooncake project. Repository and documentation (Transfer Engine, Store, traces). https://github.com/kvcache-ai/Mooncake · https://kvcache-ai.github.io/Mooncake/
6. NVIDIA Dynamo documentation. Disaggregated serving (dev). https://docs.nvidia.com/dynamo/dev/knowledge-base/concepts/system-architecture/disaggregated-serving
7. Microsoft. Azure LLM Inference Trace (AzurePublicDataset). https://github.com/Azure/AzurePublicDataset
8. Splitwise event simulator (open source). https://github.com/Mutinifni/splitwise-sim