Inference

Inference

FlashAttention Primer: IO-Aware Attention, from FA1 to FA4

FlashAttention Primer: IO-Aware Attention, from FA1 to FA4 The attention layer is the main bottleneck when scaling Transformers to long sequences: its time and memory cost grow quadratically with sequence length. FlashAttention is a family of algorithms that keeps attention exact while making it fast and memory-efficient, by treating GPU memory traffic — not just FLOPs — as the resource to optimize. This primer walks through the core ideas with original diagrams and derivations: the GPU memory hierarchy, tiling, online softmax, backward recomputation, then the FA1 → FA2 → FA3 → FA4 timeline, the difference between the standalone flash-attn package and PyTorch’s scaled_dot_product_attention, and the hardware caveats that matter in practice.
Splitting the LLM in Two: Prefill/Decode Disaggregation for SLO-Grade Serving

Splitting the LLM in Two: Prefill/Decode Disaggregation for SLO-Grade Serving A single serving instance that mixes prompt processing with token generation is simple, and for many deployments it is the right call. But once you must meet latency targets at a sustained request rate, the two phases of an LLM request start fighting each other on the same GPUs. Prefill/decode disaggregation resolves that fight by splitting the phases onto separate pools of machines, connected by a fast KV-cache handoff. This article explains what the disaggregation actually buys — and what it does not — based on the primary systems literature (DistServe, Splitwise, Mooncake) and the current official documentation of production engines (vLLM, NVIDIA Dynamo).
SGLang: From Frontend DSL to the SRT Serving Runtime

SGLang: From Frontend DSL to the SRT Serving Runtime SGLang is an open-source serving framework for large language and multimodal models, developed under the LMSYS organization and published as sgl-project/sglang under the Apache-2.0 license. As of 2026-08-09 the repository has about 31.6k stars, and the latest release is v0.5.17 (2026-08-08). One sentence of framing matters more than any of these numbers: SGLang is two systems — a Python-embedded frontend language for writing “LLM programs,“ and a serving runtime originally called SRT (SGLang Runtime). Understanding the boundary between them explains both the project’s history and its current position among inference engines.
Anatomy of the vLLM Engine

Anatomy of the vLLM Engine vLLM is the most widely deployed open-source engine for serving large language models, and its core ideas have been quietly copied by nearly every serving framework that came after it. This article walks through the engine from the inside out: the prefill/decode execution model and its latency metrics, the paged KV cache (PagedAttention), the unified scheduler of the current V1 engine, preemption and prefix caching, and the Prometheus metrics vLLM exposes for production monitoring. It closes with the boundaries against neighboring systems — FlashAttention, SGLang, and disaggregated prefill — and an honest reproduction guide for a single TITAN RTX.
用 KV 缓存量化解锁长文本生成

用 KV 缓存量化解锁长文本生成 很高兴和大家分享 Hugging Face 的一项新功能: KV 缓存量化 ，它能够把你的语言模型的速度提升到一个新水平。 太长不看版: KV 缓存量化可在最小化对生成质量的影响的条件下，减少 LLM 在长文本生成场景下的内存使用量，从而在内存效率和生成速度之间提供可定制的权衡。 你是否曾尝试过用语言模型生成很长的文本，却因为内存不足而望洋兴叹？随着语言模型的尺寸和能力不断增长，支持生成更长的文本意味着内存蚕食的真正开始。于是，磨难也随之而来了，尤其是当你的系统资源有限时。而这也正是 KV 缓存量化的用武之地。 KV 缓存量化到底是什么？如果你不熟悉这个术语，没关系！我们拆成两部分来理解: KV 缓存 和 量化 。 键值缓存或 KV 缓存是一种优化自回归模型生成速度的重要方法。自回归模型需要逐个预测下一个生成词元，这一过程可能会很慢，因为模型一次只能生成一个词元，且每个新预测都依赖于先前的生成。也就是说，要预测第 1000 个生成词元，你需要综合前 999 个词元的信息，模型通过对这些词元的表征使用矩阵乘法来完成对上文信息的抽取。等到要预测第 1001 个词元时，你仍然需要前 999 个词元的相同信息，同时还还需要第 1000 个词元的信息。这就是键值缓存的用武之地，其存储了先前词元的计算结果以便在后续生成中重用，而无需重新计算。 具体来讲，键值缓存充当自回归生成模型的内存库，模型把先前词元的自注意力层算得的键值对存于此处。在 transformer 架构中，自注意力层通过将查询与键相乘以计算注意力分数，并由此生成值向量的加权矩阵。存储了这些信息后，模型无需冗余计算，而仅需直接从缓存中检索先前词元的键和值。下图直观地解释了键值缓存功能，当计算第 K+1 个词元的注意力分数时，我们不需要重新计算所有先前词元的键和值，而仅需从缓存中取出它们并串接至当前向量。该做法可以让文本生成更快、更高效。 下一个名词是量化，它是个时髦词，主要用于降低数值的精度以节省内存。量化时，每个数值都会被舍入或截断以转换至低精度格式，这可能会导致信息丢失。然而，仔细选择量化参数和技术可以最大限度地减少这种损失，同时仍取得令人满意的性能。量化方法多种多样，如果你想知道更多信息以更深入了解量化世界，可查阅我们 之前的博文 。 有一利必有一弊，KV 缓存能够加速自回归生成，但在文本长度或者 batch size 变大时，它也随之带来了内存瓶颈。估算一下，当用 7B Llama-2 模型处理 10000 个词元的输入时，我们需要多少内存来存储 KV 缓存。存储一个词元的 KV 缓存所需的内存大致为 2 * 2 * 层数 * 键值抽头数 * 每抽头的维度 ，其中第一个 2 表示键和值，第二个 2 是我们需要的字节数 (假设模型加载精度为 float16 )。因此，如果上下文长度为 10000 词元，仅键值缓存所需的内存我们就要:
Font
Shadow
Filter
Radius
Theme color