Anatomy of the vLLM Engine

vLLM is the most widely deployed open-source engine for serving large language models, and its core ideas have been quietly copied by nearly every serving framework that came after it. This article walks through the engine from the inside out: the prefill/decode execution model and its latency metrics, the paged KV cache (PagedAttention), the unified scheduler of the current V1 engine, preemption and prefix caching, and the Prometheus metrics vLLM exposes for production monitoring. It closes with the boundaries against neighboring systems — FlashAttention, SGLang, and disaggregated prefill — and an honest reproduction guide for a single TITAN RTX.

Everything here is cited to primary sources (the PagedAttention paper, the vLLM documentation and repository, and the papers behind the adjacent systems), current as of 2026-08-09. A dated version note appears at the end. No benchmark in this article was run by the author: every performance figure is quoted from the source that measured it.

Prefill, decode, and the three latencies#

Generating text with a transformer is autoregressive: the model produces one token at a time, and each token attends to everything before it. Serving systems cut this process into two phases with very different shapes:

• Prefill processes the prompt. A prompt of $P$ tokens is run through the model in a single forward pass (or a few), in parallel, and the keys and values of every prompt token are computed and written into the KV cache. Prefill is compute-heavy and runs the whole model for each layer.
• Decode generates output tokens one at a time. Each step runs the model once for the newest token only, reading the cached keys/values of all previous tokens from the KV cache. Decode is memory-bandwidth-heavy: per step it recomputes almost no attention state, but it must stream the entire KV cache through the GPU.

The two phases produce the three latency numbers that every serving benchmark quotes:

• TTFT — time to first token: from request submission (vLLM measures from scheduling) until the first generated token is returned. It is dominated by prefill.
• ITL — inter-token latency: the time between successive output tokens during decode, i.e. the per-step latency of the decode phase.
• TPOT — time per output token: the average time to produce one output token, computed over the decode phase. vLLM’s reference Grafana dashboard treats its vllm:inter_token_latency_seconds metric as TPOT, so in practice the two are used interchangeably.

Figure 1 puts them on a timeline. A request is not interactive during TTFT, so interactive users care most about TTFT; throughput-focused workloads care about ITL/TPOT and how many requests decode concurrently.

request arrives          first token                last token
     |                       |                          |
     |<------ prefill ------>|<----- decode (one token per step) ----->|
     |<------- TTFT -------->|
                             |<-- ITL -->|<-- ITL -->|<-- ITL -->|
                             |<---------- generation time ---------->|
     |<------------------------ e2e request latency ----------------->|

Figure 1 — TTFT, ITL and TPOT on the request timeline. TTFT covers prefill; ITL/TPOT describe the decode phase. Definitions follow vLLM’s per-request metrics documentation.

How much memory does the KV cache need?#

Attention computes, for each token, a query that is compared against the keys of all previous tokens; the resulting weights are applied to the values. The keys and values of every processed token must be retained for the whole lifetime of the sequence, because every later token attends to them. That store is the KV cache, and it is the memory problem at the heart of LLM serving.

The size of the KV cache per token, at a given precision, is:

bytes per token = 2 (K and V) x bytes_per_element x num_layers x num_kv_heads x head_dim

The first factor of 2 is the key-plus-value pair; the second is the element size (2 bytes for FP16, 1 byte for FP8). With grouped-query attention (GQA) the number of KV heads is smaller than the number of query heads, which is exactly why GQA exists: it divides KV cache memory by the group size.

Model	Layers	KV heads	Head dim	KV bytes/token (FP16)	8k-token context
Llama-2-7B (MHA)	32	32	128	512 KiB	4 GiB
Llama-3.1-8B (GQA)	32	8	128	128 KiB	1 GiB
Llama-3.1-70B (GQA)	80	8	128	320 KiB	2.5 GiB

Table 1 — KV cache size per token at FP16 for three model configurations (layer/KV-head/head-dim counts from the Llama 3 model paper). MHA vs. GQA explains the 4x difference between the two 7–8B models. A 128k-token context for Llama-3.1-8B is 16 GiB of KV cache alone — as much as the model’s weights.

The KV cache is also dynamic: it grows one token at a time as decode proceeds, it is unknown in advance how long each sequence will be, and requests arrive and finish continuously. A serving engine must allocate memory for a quantity that is large, continuously growing, and unpredictable. That is the problem PagedAttention solves.

PagedAttention: paging for the KV cache#

The original PagedAttention design, described in the SOSP 2023 paper Efficient Memory Management for Large Language Model Serving with PagedAttention (Kwon et al., arXiv:2309.06180), observes that existing systems allocated each request’s KV cache as one contiguous chunk, sized for the maximum possible sequence length. The paper’s profiling (their Figure 2) found that only 20.4–38.2% of KV cache memory held actual token states in existing systems: the rest was wasted by over-reservation, internal fragmentation from over-provisioning, and external fragmentation of the allocator.

The fix is borrowed from operating systems: treat the KV cache as paged virtual memory.

• The cache is divided into fixed-size physical blocks — vLLM’s default block holds 16 tokens (CacheConfig.DEFAULT_BLOCK_SIZE).
• Each sequence owns a logical sequence of blocks; a per-sequence block table maps logical block index → physical block number.
• Physical blocks are allocated on demand as tokens are produced, so a sequence’s blocks are scattered across the cache rather than contiguous. No block is reserved for tokens that may never arrive.

Figure 2 shows the layout. The consequence is near-zero memory waste: the only unused space is the partial last block of each sequence, and a block is only allocated when it is actually needed.

GPU KV cache, 12 physical blocks (16 tokens each):

  before serving:      [0] [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11]
                       all free

  after two sequences (A = 40 prompt tokens, B = 20):

                       [0]=A t0-15   [1]=A t16-31  [2]=A t32-39
                       [3]=B t0-15   [4]=B t16-19  [5..11]=free

  block tables:        A: [0, 1, 2]          B: [3, 4]

Figure 2 — Paged KV cache. Each sequence’s block table maps logical blocks to non-contiguous physical blocks. Blocks 0 and 1 are full and could be shared with another request that has the same prefix; the freed slots 5–11 can be reused by any sequence, eliminating external fragmentation.

Two more properties make paging pay off:

• Sharing. Because a block is a pure function of its tokens, two sequences with the same prefix can point at the same physical blocks. vLLM’s original use was parallel sampling (n > 1) and beam search, where several outputs share one prompt.
• Copy-on-write (CoW). Shared blocks are read-only. When a sequence needs to write a new token into a block that other sequences still reference, the allocator duplicates the block and rewires only the writer’s block table; the other sequences keep the original. The paper describes exactly this: “copy-on-write mechanism at the block granularity for the physical blocks that need modification by multiple sequences, similar to the copy-on-write technique in OS virtual memory."

The paper’s evaluation (on A100 GPUs, with the models and workloads of its time) reports 2–4x higher throughput than state-of-the-art serving systems (FasterTransformer, Orca) at the same latency, with larger gains for longer sequences and larger models.

One historical note matters. The official vLLM documentation still contains a Paged Attention design document, but it is explicitly marked: “This is a historical document based on the original paper… It no longer describes the code used in vLLM today." The block-table idea survives; the current V1 engine implements it in a KV cache manager with a pre-allocated block pool, reference counts, and an append-only block table per request. When you read old blog posts about vLLM, keep in mind they describe the v0-era code.

V1: one scheduler for prefill and decode#

The current engine, V1, was introduced in the v0.8 series (announced January 2025) and became the default engine during the v0.9–v0.10 releases in 2025; the v0.10.0 release notes announce the start of V0 codebase cleanup, and as of v0.26.0 V1 is the engine. Architecturally, V1 splits serving into separate processes: one or more API server processes (HTTP, tokenization, streaming), one engine core process per data-parallel rank (the scheduler and KV cache manager), and one GPU worker process per GPU.

The scheduler is the interesting part. V0 scheduled prefill and decode with separate policies; V1 removes the distinction entirely. As the V1 announcement puts it, scheduling decisions are a simple dictionary, {request_id: num_tokens}: each step, the scheduler decides how many tokens each request contributes to the next forward pass. A request in prefill contributes prompt tokens (possibly a chunk of them); a request in decode contributes exactly one output token. The same mechanism covers chunked prefill, prefix caching, and speculative decoding — they are just different ways of choosing the numbers.

This buys two behaviors that were separate features in v0:

• Continuous batching. The running batch is re-formed every step: requests that finish leave immediately, waiting requests join as soon as budget and KV blocks allow. This is the iteration-level scheduling introduced by Orca (Yu et al., OSDI 2022); vLLM describes itself as a continuous batching engine.
• Chunked prefill. A long prompt is split into chunks that fit the step’s token budget, and prefill chunks are interleaved with decodes in the same forward pass. Without chunking, one long prompt would occupy the GPU for its whole prefill, and every waiting request’s TTFT would grow accordingly.

Figure 3 shows the result: prefill and decode share steps, and requests enter and leave the batch independently.

step:          1        2        3        4        5        6        7
R1 (40 tok)  [P 0-15] [P 16-31][P 32-39][D]      [D]      [D]      [done]
R2 (20 tok)  [wait]   [P 0-15] [P 16-19][D]      [D]      [done]
R3 (16 tok)  [wait]   [wait]   [P 0-15] [D]      [D]      [D]      [done]
R4 (4 tok)   [wait]   [wait]   [wait]   [P 0-3]  [D]      [D]      [D]

P x-y = prefill chunk covering prompt tokens x..y
D     = decode step (one new output token)

Figure 3 — Unified scheduling in V1. Step 4 mixes three decodes with one prefill chunk (impossible under strict phase separation). R1 finishes at step 6 and R4 keeps decoding, showing continuous batching: no request waits for the batch’s slowest member.

Preemption: RECOMPUTE#

Even with paged memory, a burst of requests can exhaust the KV cache. The scheduler then has to make room, and it does so by preempting a running request: the request’s KV blocks are freed and it is put back into the waiting queue. In V1 this is the only preemption mode — RECOMPUTE. When the preempted request is rescheduled, its prompt is recomputed from scratch; the KV blocks it needs are rebuilt, and decode resumes from where the sequence left off. The scheduler source is blunt about it: on preemption, num_computed_tokens is reset to zero.

The older v0 engine also supported SWAP preemption — copying a request’s KV blocks to CPU memory (--swap-space) — plus beam-search-style explicit block sharing via SequenceGroup. All of that is gone in V1: the metrics documentation lists vllm:num_requests_swapped and vllm:cpu_cache_usage_perc as legacy, the --swap-space flag was removed, and beam search moved out of the core engine.

Why is RECOMPUTE acceptable? Because recomputation is not necessarily wasted work: with automatic prefix caching (next section), a recomputed prompt can hit the cache and only the missing tail actually gets recomputed. The metrics documentation says exactly this — with prefix caching on by default in V1, “the preemption and recompute strategy should work better.“ Preemption still costs TTFT for the victim request and shows up in the latency metrics; it is an overload signal, not a free lunch.

Automatic prefix caching#

A large fraction of real traffic repeats prefixes: chat sessions resend the full conversation history, RAG workloads repeatedly send the same documents, agents re-prompt the same system instructions. Prefix caching stores the KV blocks of processed prompts and reuses them when a new request matches.

vLLM’s implementation is hash-based. Each full block is identified by a hash of three components (from the V1 KV cache manager design):

1. the hash of the parent block (its prefix),
2. the exact token IDs in the block,
3. extra hashes that make the block unique — LoRA IDs, multimodal input hashes, and an optional per-request cache_salt that isolates tenants from cache-based timing attacks.

Only complete blocks are cached; a partially filled trailing block is not. The block hash is what the scheduler looks up when a request arrives: matching prefix blocks are “computed blocks” whose KV is reused instead of recomputed. Figure 4 shows the chain.

request 1:  "The capital of France is"
            [b0: The capital of ]  [b1: France is]        <- both full, cached

request 2:  "The capital of France is Paris, and its largest city is Lyon"
            [b0] HIT (reused)  [b1] HIT (reused)
            [b2: Paris, and it] [b3: s largest city is]   <- computed fresh

block hashes:
  H(b0) = hash( parent=None, tokens="The capital of ", extra={} )
  H(b1) = hash( parent=H(b0), tokens="France is",       extra={} )

Figure 4 — Hash-chain prefix caching. A request reuses every full block whose hash chain matches; only the divergent tail is computed. Adapted from the V1 KV cache manager design document.

The cache manager tracks reference counts per block, keeps cached-but-unused blocks in a free queue, and evicts by LRU when memory is tight. V1 enables prefix caching by default — the V1 announcement reports less than 1% throughput loss even at a 0% cache hit rate in their benchmarks, which is what made the default safe; at high hit rates the same benchmarks report multi-fold throughput gains. As of v0.11 the default block hashing algorithm is SHA-256 (a collision-resistant choice); --prefix-caching-hash-algo sha256_cbor selects a reproducible, cross-version stable serialization.

Prefix caching also composes with preemption: a preempted-and-recomputed request can hit the prefix cache for its prompt, so RECOMPUTE preemption is often much cheaper than recomputing everything.

What the metrics tell you#

V1 exposes a Prometheus endpoint (/metrics) with a rich, vllm:-prefixed metric set. The metrics documentation’s mental model: server-level gauges and counters explain what the request-level histograms are doing. Table 2 lists the ones that matter most for operating a deployment.

Metric	Type	What it means
vllm:num_requests_running / _waiting	gauge	Requests in the RUNNING / WAITING scheduler states
vllm:kv_cache_usage_perc	gauge	Fraction of KV cache blocks in use (0–1)
vllm:prefix_cache_queries / vllm:prefix_cache_hits	counter	Prefix cache lookups and hits; their ratio is the hit rate
vllm:prompt_tokens_total / vllm:generation_tokens_total	counter	Cumulative prompt / generated tokens processed
vllm:request_success_total	counter	Finished requests, by finish reason
vllm:time_to_first_token_seconds	histogram	TTFT
vllm:inter_token_latency_seconds	histogram	ITL; treated as TPOT by the reference Grafana dashboard
vllm:e2e_request_latency_seconds	histogram	End-to-end request latency
vllm:request_queue_time_seconds	histogram	Time spent waiting before first scheduling
vllm:request_prefill_time_seconds / vllm:request_decode_time_seconds	histogram	Time in the prefill / decode phase

Table 2 — Key V1 metrics (names and semantics from the official metrics documentation; the reference Grafana dashboard uses a subset of these).

Internally, the engine core records per-request events — QUEUED, SCHEDULED, PREEMPTED, NEW_TOKENS — and the frontend derives the intervals: queue time, prefill time (first SCHEDULED → first NEW_TOKENS), decode time, inference time, and inter-token time. Two operational notes: a preemption during decode stretches the decode/ITL intervals of the victim (documented in the metrics design), so a spike in ITL paired with a rise in vllm:num_requests_waiting or KV-cache pressure is the signature of preemption-induced latency; and the prefix hit rate — hits/queries — tells you whether your traffic actually repeats prefixes, i.e. whether prefix caching is earning its keep.

Boundaries: FlashAttention, SGLang, and disaggregated prefill#

vLLM does not own every optimization in its stack, and several neighboring systems solve overlapping problems differently. The boundaries are worth stating precisely.

FlashAttention is a kernel, PagedAttention is a memory layout. FlashAttention (Dao et al., 2022) and FlashAttention-2 (Dao, 2023) are fused attention kernels: they tile the attention computation and use online softmax so the full score matrix is never materialized, cutting both time and memory at the arithmetic level. PagedAttention decides where KV values live and how they are indexed; FlashAttention decides how fast the attention math runs over them. vLLM treats attention backends as pluggable: FLASH_ATTN (the flash-attn package), FLASHINFER, FLASHMLA for MLA models, and a Triton attention backend, auto-selected by priority and hardware capability. That is also the reason a GPU generation matters: vLLM’s FlashAttention backend validates compute capability ≥ 8.0, FlashInfer targets sm80+, FlashAttention-3 is Hopper-only, and FlashAttention-4 targets Hopper/Blackwell — stock flash-attn explicitly supports only Ampere and newer (a separate community project covers a subset on Turing).

SGLang is an alternative system with a different cache structure. SGLang (Zheng et al., 2023) is a separate serving system that solves the same prefix-reuse problem with RadixAttention: KV blocks live in a radix tree keyed by token sequences, so shared prefixes are shared nodes, with LRU eviction and copy-on-write. vLLM V1’s hash-based block cache and SGLang’s radix tree are two designs for the same goal — reuse KV across requests with common prefixes — and both projects claim high cache hit rates under prefix-heavy workloads. They are alternatives, not components; the practical choice is usually ecosystem and workload fit, not a decisive technical gap. (The projects also share authors and ideas: both descend from the same research group lineage.)

Disaggregated prefill moves the phases to different machines. The idea, introduced by DistServe (Zhong et al., 2024), is to run prefill and decode on separate instances, connected by a KV transfer path, so each phase can be scaled and tuned independently. vLLM implements this as an experimental feature: a prefill instance and a decode instance of vLLM exchange KV blocks through a connector (NIXL, Mooncake, LMCache, and others). The documented motivations are exactly the metrics of this article: tune TTFT and ITL separately, and control tail ITL by removing prefill jobs from decode instances — the docs are explicit that disaggregated prefill does not improve throughput. Within a single instance, chunked prefill is the simpler tool for the same tail-latency problem.

Reproducing this on a TITAN RTX#

The workstation this article was written on has a NVIDIA TITAN RTX: 24 GB GDDR6 at 672 GB/s, Turing TU102, compute capability 7.5 (NVIDIA product specifications). That places it at the minimum supported architecture for vLLM (the installation docs require compute capability 7.5 or higher), which has concrete consequences:

• vLLM’s FlashAttention backend requires compute capability ≥ 8.0, FlashInfer requires sm80+, FlashAttention-3 needs Hopper. On a TITAN RTX, V1 auto-selects its Triton attention backend instead. Correctness and the metrics pipeline work; the paper’s A100/H100 kernel numbers do not transfer.
• Everything fits, barely. A Llama-3.1-8B in FP16 is about 16 GiB of weights; with vLLM’s default gpu_memory_utilization=0.92, roughly 6 GiB remains for KV cache and scratch — at 128 KiB per token (Table 1) that is on the order of 40–50k tokens of cache capacity. These are arithmetic estimates, not measurements.
• The FP16 tensor throughput NVIDIA quotes for this card (roughly 130 TFLOPS) is a peak kernel number that end-to-end serving will never approach; do not use it to predict TTFT or ITL.

A honest reproduction of the qualitative behavior takes minutes and no benchmark framework: run vllm serve with a small model (current releases default to Qwen3-0.6B), then

1. TTFT/ITL: watch vllm:time_to_first_token_seconds and vllm:inter_token_latency_seconds on /metrics while sending requests of different prompt lengths. Expect TTFT to scale with prompt size; ITL should stay roughly flat per request.
2. Prefix caching: send the same long prompt repeatedly. vllm:prefix_cache_hits should climb and prefill time should collapse after the first request.
3. Preemption: raise concurrency until vllm:kv_cache_usage_perc saturates; vllm:num_requests_waiting rises and the ITL histogram of preempted requests stretches.

What you will not get from this card is a number comparable to vendor or paper benchmarks — the workloads, GPUs, and software versions differ. If you need a publishable number, run the official benchmarking tools on the hardware you intend to ship, and say so in the methodology. Every figure cited in this article was measured by its source’s authors on their hardware; none were reproduced here.

Version note#

This article describes vLLM as of 2026-08-09: the latest release is v0.26.0 (published 2026-07-27). V1 is the engine; it was introduced in the v0.8 series (announced January 2025), became the default during the v0.9–v0.10 releases, and the V0 codebase cleanup began with v0.10. The official documentation marks the original Paged Attention design document as historical. Flag names, metric names, and defaults change between releases — for example, --prefix-caching-hash-algo and the SHA-256 default landed in v0.11. Verify against the documentation of the version you deploy.

Bibliography#

1. W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, I. Stoica. Efficient Memory Management for Large Language Model Serving with PagedAttention. SOSP 2023. arXiv:2309.06180. https://arxiv.org/abs/2309.06180
2. vLLM project — official documentation (Architecture Overview; Paged Attention [historical]; Prefix Caching; Metrics; Automatic Prefix Caching; Disaggregated Prefill; Per-request Metrics; Installation). https://docs.vllm.ai/en/latest/
3. vLLM project — source repository, release v0.26.0. https://github.com/vllm-project/vllm
4. vLLM Team. vLLM V1: A Major Upgrade to vLLM’s Core Architecture. vLLM Blog, 2025-01-27. https://blog.vllm.ai/2025/01/27/v1-alpha-release.html
5. T. Dao, D. Fu, S. Ermon, A. Rudra, C. Ré. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. NeurIPS 2022. arXiv:2205.14135.
6. T. Dao. FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning. ICLR 2024. arXiv:2307.08691.
7. L. Zheng, L. Yin, Z. Xie, C. Sun, J. Huang, C. H. Yu, S. Cao, C. Kozyrakis, I. Stoica, J. E. Gonzalez, C. Barrett, Y. Sheng. SGLang: Efficient Execution of Structured Language Model Programs. NeurIPS 2024. arXiv:2312.07104.
8. Y. Zhong, S. Liu, J. Chen, J. Hu, Y. Zhu, X. Liu, X. Jin, H. Zhang. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. OSDI 2024. arXiv:2401.09670.
9. G.-I. Yu, J. S. Jeong, G.-W. Kim, S. Kim, B.-G. Chun. Orca: A Distributed Serving System for Transformer-Based Generative Models. OSDI 2022. arXiv:2109.14456.
10. A. Grattafiori et al. The Llama 3 Herd of Models. arXiv:2407.21783. (Layer/KV-head/head-dim configurations used in Table 1.)
11. NVIDIA. NVIDIA TITAN RTX product specifications. https://www.nvidia.com/en-us/geforce/graphics-cards/titan-rtx/
12. Dao-AILab. flash-attention repository README (supported-GPU statement). https://github.com/Dao-AILab/flash-attention