SGLang: From Frontend DSL to the SRT Serving Runtime

SGLang is an open-source serving framework for large language and multimodal models, developed under the LMSYS organization and published as sgl-project/sglang under the Apache-2.0 license. As of 2026-08-09 the repository has about 31.6k stars, and the latest release is v0.5.17 (2026-08-08). One sentence of framing matters more than any of these numbers: SGLang is two systems — a Python-embedded frontend language for writing “LLM programs,“ and a serving runtime originally called SRT (SGLang Runtime). Understanding the boundary between them explains both the project’s history and its current position among inference engines.

This article is a concept-level tour: frontend versus runtime, RadixAttention and prefix reuse, cache-aware scheduling, structured generation, parallelism, and where the comparison boundary with vLLM sits today. Performance numbers are quoted from the primary sources (the NeurIPS 2024 paper, official release blogs, and official documentation) with their dates, hardware, and baseline versions; none of them were reproduced locally.

Two Faces: A Frontend Language and a Runtime#

The paper SGLang: Efficient Execution of Structured Language Model Programs (arXiv:2312.07104, first posted 2023-12-12, revised 2024-06-06, presented as a poster at NeurIPS 2024) opens with a design split that remains intact today:

1. Frontend — a domain-specific language embedded in Python for expressing prompts with generation calls, branching, and parallelism.
2. Runtime (SRT) — a scheduler, a radix-tree KV cache manager, and a model runner that execute those programs (or plain serving requests) efficiently.

The paper is explicit that the two parts can function independently: the frontend can target remote API models, and the runtime serves ordinary requests with no DSL involvement. That independence is the key to reading SGLang’s evolution — the runtime became a general-purpose serving engine, while the frontend became one (optional) face of the project.

The Frontend: A Python-Embedded DSL#

The frontend language is ordinary Python decorated with @function. Inside a function, a State object is built by appending message blocks (s += user(...), s += assistant(...), s += system(...)) and generation calls. The core primitives, per the paper and the current frontend tutorial in the official docs:

• gen("name", ...) — generate text, bound to a variable; supports max_tokens, stop, temperature, and constraints such as choices=[...] or regex=....
• select(...) — choose among options (the paper’s terminology for constrained choice).
• s += ... / extend — append to the conversation state.
• fork / join — launch parallel branches of the program.
• image(...) — attach images for multimodal models.

Because gen is non-blocking, several generation calls can run concurrently within one program. Two execution modes are documented: an interpreter mode with an async stream executor that runs the program’s parallelism directly, and a compiler mode that traces the program into a graph. A “frontend hint” sends the common prefix of forked branches to the runtime first, which lets the radix cache absorb it (more below). The same program can run against a local SRT server or against API backends (OpenAI, Anthropic, and others are supported via the runtime endpoint abstraction) — a design that predates and parallels today’s agent frameworks.

A minimal example from the official frontend tutorial:

from sglang import function, user, assistant, gen

@function
def basic_qa(s, question):
    s += user(question)
    s += assistant(gen("answer", max_tokens=512))

state = basic_qa("List 3 countries and their capitals.")
print(state["answer"])

The Runtime: SRT#

The runtime side — the scheduler plus radix-tree KV cache plus model runner — is what most people mean by “SGLang” today. The current project exposes three surfaces:

• an OpenAI-compatible API (/v1/chat/completions, /v1/completions, plus Anthropic- and Ollama-compatible endpoints), so standard OpenAI clients work unchanged;
• a native /generate endpoint with direct sampling_params; and
• an offline sgl.Engine for in-process batch inference without an HTTP server.

The official quickstart launches a server with:

python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct --host 0.0.0.0 --port 30000

and waits for the banner The server is fired up and ready to roll!. The README’s feature list summarizes the runtime’s current scope: RadixAttention prefix caching, a zero-overhead CPU scheduler, prefill–decode disaggregation, speculative decoding, continuous batching, paged attention, tensor/pipeline/expert/data parallelism, structured outputs, chunked prefill, quantization, and multi-LoRA batching.

RadixAttention: Prefix Reuse as a First-Class Concern#

Most serving workloads re-send long shared prefixes: multi-turn chat history, few-shot exemplars, system prompts, retrieval-augmented contexts. Without caching, every request recomputes the key–value activations of those tokens. RadixAttention, the paper’s headline optimization, treats the KV cache as a radix tree so that shared prefixes are computed once and referenced many times.

Prefix Reuse with a Radix Tree#

The KV cache is organized as a tree in which each node holds the KV tensors of a contiguous run of tokens (paged into 1-token or multi-token pages); the path from root to leaf is a full request prefix. Requests that share a prefix share the corresponding nodes. Key mechanics from the paper (Section 3):

• Eviction is LRU on leaves first, so the least-recently-used leaf subtree is freed before anything an active request depends on.
• Reference counting protects in-flight requests — a node referenced by the running batch is not evicted.
• Cache and running requests share one memory pool, so idle cache memory automatically shrinks under load and regrows when capacity allows.
• Cache-aware scheduling picks the request with the longest shared prefix first, which is approximately a depth-first traversal of the tree. The paper’s Theorem 3.1 states this DFS order achieves the optimal offline cache hit rate when the cache holds at most the maximum request length.
• Overhead is small when there are no hits: the paper measures under 0.3% throughput loss on a ShareGPT workload with an empty cache.

The mechanism also covers modalities: image tokens are reused by hashing the input image (identical images in a batch or across requests skip re-encoding), and data-parallel deployments route by a router-level meta-tree so requests land on the worker that already holds their prefix.

The first announcement blog (2024-01-17) reported up to 5× throughput from RadixAttention on workloads with shared prefixes — the paper’s own headline claim is up to 6.4× throughput versus state-of-the-art inference systems of its time across agent control, logical reasoning, few-shot benchmarks, JSON decoding, RAG pipelines, and multi-turn chat. Both figures are vendor claims from 2024-era baselines, not independent measurements.

From the 2024 Paper to Today’s Cache Stack#

The radix tree in the paper was GPU-memory-only and request-scoped. Two documented evolutions define the 2026 state:

• Session-aware radix caching (implemented in UnifiedRadixCache). Applications pass a session_id on each request; finished requests register their reusable cache leaves under that session, and /close_session releases them. Eviction prefers unreferenced nodes first, then session-referenced nodes, then LRU, with a component cascade for hybrid models: evicting a Full-attention node also drops its sliding-window-attention (SWA) and Mamba data; evicting SWA drops Mamba data only. Sessions are soft references, not memory pins. Enable with SGLANG_ENABLE_UNIFIED_RADIX_TREE=1 plus --enable-session-radix-cache.
• HiCache (announced 2025-09-10) extends caching beyond GPU memory in the style of a CPU cache hierarchy: GPU memory is L1, host DRAM is L2, and distributed storage is L3 (with integrations such as Mooncake, 3FS, NIXL, and AIBrix KVCache). A HiRadixTree records where each KV span lives, and a local-match → prefetch → write-back workflow keeps reuse working across tiers.

Cache-Aware Scheduling Beyond the Tree#

Two scheduling advances from the v0.4 release (2024-12-04) are worth understanding separately from RadixAttention itself:

• Zero-overhead batch scheduler. CPU-side work — batch formation, memory allocation, prefix matching — is overlapped with GPU computation by running the scheduler one batch ahead. The v0.4 announcement reports 1.1× throughput versus the previous version and 1.3× versus other state-of-the-art baselines (small models and large tensor-parallel sizes benefit most), verified with Nsight profiling. It is on by default; the current docs expose the A/B switch as --disable-overlap-schedule.
• Cache-aware load balancer (sglang-router). For data-parallel deployments, a router keeps an approximate radix tree of each worker’s cache and routes each request to the worker with the best prefix match, without cross-worker synchronization. The v0.4 announcement reports up to 1.9× throughput and 3.8× cache-hit-rate improvement (8× A100-80GB, a generated-shared-prefix workload, v0.4 versus v0.3). It ships as a standalone Rust package (pip install sglang-router) usable as a drop-in for --dp-size.

Related to both: chunked prefill (--chunked-prefill-size) splits long prompts into smaller pieces so a batch can mix prefill and decode work instead of stalling on one giant prompt — important for long-context and PD-disaggregated deployments.

Structured Generation#

“Structured outputs” means constraining decoding so the output is guaranteed to match a grammar — a JSON schema, a regular expression, or an EBNF grammar. This is the other paper-era optimization, and it has changed backends since.

Compressed Finite-State Machines (Paper Era)#

The paper’s approach (Section 4) compiles a regex/JSON constraint into a finite-state machine, then compresses it by merging singular-transition edges into jumps, so the decoder can emit several tokens in a single forward pass (“Jump Forward”) and re-tokenize with the original tokenizer. The announcement blog (2024-02-05) reports 3× faster JSON decoding versus baselines of that time.

XGrammar and Today’s Backends#

Since v0.4, the default grammar backend is XGrammar (from the MLC/Apache TVM community); the server flag --grammar-backend accepts xgrammar, outlines, llguidance, or none. Constraints are passed either through the OpenAI-compatible response_format (e.g. {"type": "json_schema", "json_schema": {...}}) or via extra_body/sampling parameters (regex=..., ebnf=...), including the structural_tag format for tool-call style outputs. XGrammar’s own paper (arXiv:2411.15100, November 2024, MLSys 2025) reports up to 100× speedup in grammar execution versus prior constrained-decoding libraries and near-zero overhead end-to-end; the v0.4 announcement reports SGLang + XGrammar up to 10× faster JSON decoding than other open-source solutions at the time. Again: vendor-published numbers on 2024 baselines.

Parallelism: TP, PP, DP, EP, and PD Disaggregation#

Conceptually, the runtime composes four classic axes, plus disaggregation:

• Tensor parallelism (TP) — --tp-size (aliased --tensor-parallel-size): split each layer’s weights across GPUs, with all-reduce per layer. The default for fitting large models on one node.
• Pipeline parallelism (PP) — --pp-size: split layers across GPUs with micro-batch pipelining (the docs document async micro-batching, e.g. --pp-async-batch-depth), for models too large for one node.
• Data parallelism (DP) — --dp-size: replicate the model and partition requests across workers; this is where the cache-aware router earns its keep, because naive round-robin destroys prefix locality.
• Expert parallelism (EP) — --ep-size: for MoE models, distribute experts across GPUs and use all-to-all communication to route tokens to the right experts. The all-to-all backend is selectable via --moe-a2a-backend (deepep, mooncake, nixl, mori, pplx, and others in the current docs), and expert load balancing is available via --enable-eplb.
• Attention-specific DP (--enable-dp-attention) — a MoE-flavored hybrid: data-parallel attention with tensor-parallel FFN, aimed at MLA models with a single KV head (DeepSeek-style) where naive TP duplicates the KV cache. The v0.4 announcement reports 1.9× decoding throughput versus v0.3 on 8× H100-80GB (DeepSeek-Coder-V2).
• Prefill–decode (PD) disaggregation — --disaggregation-mode prefill|decode splits the two phases onto separate servers so each can be scaled and scheduled independently; the KV cache moves between them via a transfer backend (--disaggregation-transfer-backend, default mooncake, also nixl, ascend, mori, …), with optional radix caching on the decode side (--disaggregation-decode-enable-radix-cache). Multi-node variants are covered in the deployment docs.

The official milestones show this axis growing continuously: PD + large-scale EP on 96 H100s (2025-05-05, reporting 2.7× decode gains in the GB200 Part I blog of 2025-06-16, and 3.8× prefill / 4.8× decode in Part II of 2025-09-25), a JAX/TPU backend (2025-10-29), and a GB300 write-up claiming 25× on NVIDIA GB300 NVL72 (2026-02-20). All figures are official-blog claims with their own hardware and versions — useful as trend evidence, not as portable benchmarks.

Where SGLang and vLLM Draw the Line Today#

vLLM (vllm-project/vllm) is SGLang’s closest neighbor: a Python, Apache-2.0 serving engine with OpenAI-compatible APIs, paged KV cache management, continuous batching, TP/PP/DP/EP/context parallelism, and pluggable grammar backends (it also integrates XGrammar, Outlines, and guidance). As of 2026-08-09 vLLM has about 88.6k GitHub stars, versus SGLang’s 31.6k. vLLM originates from the PagedAttention paper (arXiv:2309.06180, SOSP 2023), which reported 2–4× throughput over FasterTransformer and Orca on 2023 hardware.

The boundary is not in the engines’ feature lists, which converged; it is in the project’s outer surface:

• vLLM is engine-only. It serves requests through OpenAI-compatible, Anthropic-compatible, and gRPC APIs and an offline engine. There is no frontend programming language: no @function programs, no fork/join parallelism in the request definition, no interpreter/compiler duality.
• SGLang is engine-plus-frontend. The same runtime can be used purely as an OpenAI-compatible server, but the DSL layer remains a supported, documented way to express multi-step generation programs, and the frontend can target non-SGLang backends (OpenAI/Anthropic APIs) as well.
• The prefix-caching ideas share a lineage. vLLM’s automatic prefix caching (RFC issue #2614, opened 2024-01-26) explicitly models its block eviction policy on RadixAttention — reference counts first, then LRU, then prefix length. So “radix-style” caching is now common industry practice, not a differentiator; the differentiator is how each project builds on it (SGLang’s session-aware tiers and HiCache, vLLM’s hash-table design).
• Serving benchmarks are engine-versus-engine and date-stamped. Both projects publish throughput comparisons (SGLang’s v0.2 Llama-3 blog, v0.4 numbers above; vLLM’s own benchmark suite). These are vendor claims on specific hardware, models, and versions — the honest way to compare today is to run the official benchmarking tools (below) on your own hardware.

A practical summary of the boundary:

A Dated Evolution Timeline#

Table 1 — SGLang milestones, each with its primary source (paper, official release blog, or docs):

Reproducibility Checklist#

The commands below are taken verbatim from the official quickstart, frontend tutorial, structured-outputs, and bench-serving pages (docs.sglang.io, retrieved 2026-08-09). We did not execute them in this environment — the checklist is a path to reproduce the article’s claims, and the numbers in the previous sections require the specific hardware and versions cited there.

1. Install (official quickstart):

   pip install --upgrade pip
   pip install uv
   uv pip install --prerelease=allow sglang
   

   (Docker alternative: lmsysorg/sglang:latest from Docker Hub.)

2. Launch a server and wait for The server is fired up and ready to roll!:

   python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct --host 0.0.0.0 --port 30000
   

3. Smoke-test the OpenAI-compatible API:

   curl http://localhost:30000/v1/chat/completions \
   -H "Content-Type: application/json" \
   -d '{"model": "qwen/qwen2.5-0.5b-instruct",
       "messages": [{"role": "user", "content": "What is the capital of France?"}]}'
   

   Or the native endpoint: POST /generate with {"text": "...", "sampling_params": {"temperature": 0, "max_new_tokens": 32}}.

4. Offline engine (no server):

   import sglang as sgl
   llm = sgl.Engine(model_path="qwen/qwen2.5-0.5b-instruct")
   outputs = llm.generate(["Hello, my name is"], sampling_params={"temperature": 0.8, "top_p": 0.95})
   llm.shutdown()
   

5. Throughput benchmark (official bench-serving guide), against a running server:

   python3 -m sglang.bench_serving \
   --backend sglang --host 127.0.0.1 --port 30000 \
   --model meta-llama/Llama-3.1-8B-Instruct \
   --dataset-name random --random-input-len 1024 --random-output-len 1024 \
   --num-prompts 1000
   

   To compare against vLLM on the same hardware, run the same tool with --backend vllm --base-url http://127.0.0.1:8000 (and use --flush-cache so warm-cache effects don’t skew the comparison).

6. A/B the scheduler: relaunch with --disable-overlap-schedule and repeat step 5.

7. Structured output (official structured-outputs page):

   client.chat.completions.create(
    model="meta-llama/Meta-Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": "Give me the capital of France in JSON."}],
    temperature=0, max_tokens=128,
    response_format={"type": "json_schema",
                     "json_schema": {"name": "foo", "schema": {...}}},
   )
   

8. Frontend DSL: run the @function example from the frontend tutorial against the server from step 2 (via RuntimeEndpoint).

If any flag or endpoint differs on your install, pin the version first — the CLI surface evolves quickly (for example, the v0.4 blog used --disable-overlap; the 2026 docs use --disable-overlap-schedule).

Caveats: How to Read the Numbers#

Every quantitative claim in this article is a cited claim with a dated baseline, per the sources in the bibliography — not an independent measurement:

• Paper (2024-era baselines): up to 6.4× throughput; <0.3% radix-cache overhead with no hits; optimal offline hit rate under DFS scheduling (Theorem 3.1).
• 2024 release blogs: 5× (RadixAttention, 2024-01), 3× JSON decoding (2024-02), 7× DeepSeek MLA (v0.3, 2024-09), 1.1×/1.3× scheduler, 1.9×/3.8× router, 1.9× DP-attention decode, 10× XGrammar JSON (v0.4, 2024-12).
• 2025–2026 hardware blogs: 2.7× decode (GB200 Part I, 2025-06), 3.8× prefill / 4.8× decode (GB200 Part II, 2025-09), 25× (GB300, 2026-02) — each on specific NVIDIA hardware with specific models and versions.
• XGrammar paper: up to 100× grammar-execution speedup (arXiv:2411.15100, 2024-11).

Hardware, model, and software versions move faster than prose: treat the numbers as directional, and re-measure on your own GPU with the checklist above.

References#

1. Lianmin Zheng et al. SGLang: Efficient Execution of Structured Language Model Programs. arXiv:2312.07104 (v2 2024-06-06); presented at NeurIPS 2024 (poster 94872). https://arxiv.org/abs/2312.07104
2. sgl-project/sglang — official repository (Apache-2.0). https://github.com/sgl-project/sglang
3. SGLang official documentation (docs.sglang.io), retrieved 2026-08-09: Quickstart; Frontend Language; Structured Outputs; Session-Aware Radix Cache; HiCache System Design; Server Arguments; Bench Serving Guide. https://docs.sglang.io/
4. The SGLang Team. SGLang v0.4: Zero-Overhead Batch Scheduler, Cache-Aware Load Balancer, Faster Structured Outputs. LMSYS blog, 2024-12-04. https://lmsys.org/blog/2024-12-04-sglang-v0-4/
5. SGLang release blogs: RadixAttention (2024-01-17), compressed FSM (2024-02-05), v0.2 (2024-07-25), v0.3 (2024-09-04), large-scale EP (2025-05-05), GB200 parts I/II (2025-06-16, 2025-09-25), HiCache (2025-09-10), SGLang-JAX (2025-10-29), GB300 (2026-02-20), DeepSeek-V4 (2026-04-25), DFlash/Spec V2 (2026-06-15), Kimi K3 (2026-07-27). https://lmsys.org/blog/
6. Woosuk Kwon et al. Efficient Memory Management for Large Language Model Serving with PagedAttention. arXiv:2309.06180 (SOSP 2023). https://arxiv.org/abs/2309.06180
7. vLLM RFC: Automatic Prefix Caching (issue #2614, 2024-01-26). https://github.com/vllm-project/vllm/issues/2614
8. Yixin Dong et al. XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Models. arXiv:2411.15100 (2024-11-22; MLSys 2025). https://arxiv.org/abs/2411.15100
9. vllm-project/vllm — official repository (Apache-2.0). https://github.com/vllm-project/vllm