Posts

Shipping static sites

Local preview uses nginx + prebuilt public/. Production deploy is intentionally out of band.
FlashAttention Primer: IO-Aware Attention, from FA1 to FA4

FlashAttention Primer: IO-Aware Attention, from FA1 to FA4 The attention layer is the main bottleneck when scaling Transformers to long sequences: its time and memory cost grow quadratically with sequence length. FlashAttention is a family of algorithms that keeps attention exact while making it fast and memory-efficient, by treating GPU memory traffic — not just FLOPs — as the resource to optimize. This primer walks through the core ideas with original diagrams and derivations: the GPU memory hierarchy, tiling, online softmax, backward recomputation, then the FA1 → FA2 → FA3 → FA4 timeline, the difference between the standalone flash-attn package and PyTorch’s scaled_dot_product_attention, and the hardware caveats that matter in practice.
Splitting the LLM in Two: Prefill/Decode Disaggregation for SLO-Grade Serving

Splitting the LLM in Two: Prefill/Decode Disaggregation for SLO-Grade Serving A single serving instance that mixes prompt processing with token generation is simple, and for many deployments it is the right call. But once you must meet latency targets at a sustained request rate, the two phases of an LLM request start fighting each other on the same GPUs. Prefill/decode disaggregation resolves that fight by splitting the phases onto separate pools of machines, connected by a fast KV-cache handoff. This article explains what the disaggregation actually buys — and what it does not — based on the primary systems literature (DistServe, Splitwise, Mooncake) and the current official documentation of production engines (vLLM, NVIDIA Dynamo).
SGLang: From Frontend DSL to the SRT Serving Runtime

SGLang: From Frontend DSL to the SRT Serving Runtime SGLang is an open-source serving framework for large language and multimodal models, developed under the LMSYS organization and published as sgl-project/sglang under the Apache-2.0 license. As of 2026-08-09 the repository has about 31.6k stars, and the latest release is v0.5.17 (2026-08-08). One sentence of framing matters more than any of these numbers: SGLang is two systems — a Python-embedded frontend language for writing “LLM programs,“ and a serving runtime originally called SRT (SGLang Runtime). Understanding the boundary between them explains both the project’s history and its current position among inference engines.
Anatomy of the vLLM Engine

Anatomy of the vLLM Engine vLLM is the most widely deployed open-source engine for serving large language models, and its core ideas have been quietly copied by nearly every serving framework that came after it. This article walks through the engine from the inside out: the prefill/decode execution model and its latency metrics, the paged KV cache (PagedAttention), the unified scheduler of the current V1 engine, preemption and prefix caching, and the Prometheus metrics vLLM exposes for production monitoring. It closes with the boundaries against neighboring systems — FlashAttention, SGLang, and disaggregated prefill — and an honest reproduction guide for a single TITAN RTX.
Font
Shadow
Filter
Radius
Theme color