# SGLang

SGLang is an open-source LLM serving framework with RadixAttention prefix caching, fast structured output and efficient scheduling. Learn when to use it.

## Overview

SGLang is an open-source framework for serving large language models and multimodal models at high speed. It has two parts. The runtime is a serving engine that loads models onto GPUs and answers requests through an OpenAI-compatible API, with an emphasis on reusing work across requests. The frontend is a small Python language for writing programs that make several model calls, with branching, parallel generation and constrained output, which the runtime can execute efficiently.

Its signature idea is RadixAttention. Instead of throwing away a request's cached attention state when it finishes, SGLang keeps it in a tree organised by token prefixes. When a new request starts with the same tokens, such as a shared system prompt, few-shot examples or earlier turns of a conversation, the engine reuses the cached part and only computes what is new.

An everyday picture: a library photocopier queue. If ten people each need the same first 40 pages of a report plus a few pages of their own, a sensible clerk copies the 40 pages once and hands everyone a copy, then copies only the extra pages for each person. SGLang is that clerk. Shared beginnings are computed once; only the differences cost extra.

## What it is

| Part | Purpose |
|---|---|
| SGLang Runtime | The serving engine with scheduler, memory manager and model executor |
| RadixAttention | Automatic reuse of [KV cache](/tools/kv-cache) across requests using a radix tree of prefixes |
| Continuous batching | Adds and removes requests from the running batch at each step |
| Cache-aware scheduling | Orders waiting requests to maximise prefix reuse |
| Constrained decoding | Forces output to follow a JSON schema, regular expression or grammar |
| Frontend language | Python primitives such as `gen`, `select` and `fork` for multi-call programs |
| OpenAI-compatible server | Chat, completion and embedding endpoints |
| Parallelism | Tensor, pipeline, data and expert parallelism across GPUs and nodes |
| Disaggregation | Running prefill and decode on separate workers for large deployments |
| Router | Distributes requests across engine instances with prefix awareness |

SGLang supports many open model families, including [mixture-of-experts](/tools/mixture-of-experts) models, multimodal models and embedding models, and runs on NVIDIA and AMD GPUs and other accelerators.

## Where it came from

SGLang was developed by researchers associated with LMSYS, a research group with members from UC Berkeley, Stanford and other institutions, led by authors including Lianmin Zheng and Ying Sheng. The paper "SGLang: Efficient Execution of Structured Language Model Programs" appeared on arXiv in December 2023, an LMSYS blog post introduced RadixAttention and the system in January 2024, and the paper was published at NeurIPS 2024.

The original motivation was that real applications rarely make one isolated model call. Agents, chatbots with long histories, few-shot pipelines and evaluation harnesses make many calls with heavily shared prompts, and existing engines repeated much of that work. Over 2024 and 2025 the runtime grew into a general-purpose, high-performance engine, adding fast constrained decoding, support for large mixture-of-experts models, multi-node expert parallelism, prefill and decode disaggregation, and a router for scaling across many instances. It became widely used in both research and production, and is released under the Apache 2.0 licence.

## How it works

### Prefix sharing with a radix tree

A radix tree stores sequences so that shared beginnings are stored once. SGLang keeps the KV cache of finished requests in such a tree, with each edge representing a run of tokens and the cache blocks that hold them. When a new request arrives, the engine walks the tree to find the longest matching prefix and reuses those blocks. Only the remaining tokens go through prefill. When memory is needed, the least recently used leaves are evicted first.

### Cache-aware scheduling

If many requests are waiting, running those that share prefixes together increases reuse. SGLang's scheduler takes the cached prefix length into account, while still avoiding starvation of other requests.

### Constrained decoding

For structured output, the engine restricts which tokens the model may choose at each step so the result always matches a schema or grammar. SGLang compiles these constraints so that predictable stretches, such as fixed JSON keys, can be emitted with little overhead, and integrates fast grammar back-ends.

### The frontend language

Programs describe a sequence of prompt pieces and generation calls. The runtime sees the whole structure, so it can reuse prefixes and run independent branches in parallel.

```python
import sglang as sgl

@sgl.function
def classify(s, ticket):
    s += sgl.user("Classify this ticket: " + ticket)
    s += sgl.assistant(sgl.gen("label", choices=["billing", "bug", "other"]))
```

Most production users, however, call the runtime through its OpenAI-compatible API, launched with `python -m sglang.launch_server` and a model path.

### Scaling

Large models are split across GPUs with tensor or expert parallelism. For high-volume deployments, prefill and decode can run on separate pools, and the router sends requests to instances that already hold matching prefixes.

## Key concepts and terms

| Term | Meaning |
|---|---|
| RadixAttention | KV cache reuse across requests through a prefix tree |
| Radix tree | A tree that stores shared sequence prefixes once |
| Prefix hit rate | The share of prompt tokens served from cache |
| KV cache | Stored attention keys and values for processed tokens |
| Continuous batching | Changing batch membership at every generation step |
| Constrained decoding | Restricting tokens so output follows a format |
| Grammar | A formal description of allowed output structure |
| `fork` | A frontend primitive that runs several branches from one state |
| Prefill and decode disaggregation | Separating prompt processing and token generation onto different workers |
| Expert parallelism | Spreading a mixture-of-experts model's experts across devices |
| Router | A component that balances requests across engine instances |

## A worked example

This scenario is illustrative. A software company runs an agent that answers support questions using tools.

1. Every agent call starts with the same 3,000-token prefix: instructions, tool definitions and policy text. Each conversation then adds its own history.
2. On their previous engine without automatic prefix reuse across these calls, each step re-processed most of the prefix, and time to first token averaged about 900 milliseconds under load.
3. With SGLang, the shared prefix stays in the radix tree. Measured prefix hit rate is about 85 per cent, and median time to first token falls to about 300 milliseconds on the same GPUs.
4. Tool calls must be valid JSON. With a schema constraint, malformed calls drop from about 2 per cent of steps to none, removing a retry loop.
5. As traffic grows, they run four engine instances behind the SGLang router, which sends each conversation back to the instance that holds its history.

## Strengths and limitations

**Strengths:**

- Automatic, fine-grained reuse of shared prefixes across requests.
- Fast constrained decoding for JSON and other formats.
- High throughput, including for large mixture-of-experts models.
- OpenAI-compatible API plus a frontend for multi-call programs.
- Open source with an active research and industry community.

**Limitations:**

- Gains depend on workload; little prefix sharing means smaller benefits.
- Fast development pace means changing options and defaults.
- Advanced features such as disaggregation need careful infrastructure.
- The frontend language is less widely used than the server API, so fewer examples exist.

## How it compares with alternatives

| Engine | Strength | Notes |
|---|---|---|
| SGLang | Automatic prefix reuse, structured generation, agent workloads | Strong for multi-turn and multi-call use |
| [vLLM](/tools/vllm) | Broad, high-throughput general serving | Also offers prefix caching |
| [TensorRT](/tools/tensorrt) with TensorRT-LLM | Peak performance on NVIDIA GPUs | More build steps |
| [llama.cpp](/tools/llama-cpp) | Local and edge inference | Smaller scale |
| [Ray Serve](/tools/ray-serve) | Orchestration around engines | Can run SGLang underneath |

## Practical guidance and best practices

1. Put shared content, such as instructions and tool definitions, at the start of prompts and keep it byte-for-byte identical.
2. Avoid inserting timestamps or random IDs early in prompts, which break prefix matching.
3. Monitor prefix hit rate along with latency and throughput.
4. Use constrained decoding for any output a program must parse.
5. Size GPU memory to hold the shared prefixes plus active requests.
6. Use the router with prefix awareness when running several instances.
7. Benchmark against your real traffic, not generic tests, before choosing an engine.
8. Pin versions and test upgrades in staging.

## Common mistakes

- Placing per-user details before the shared system prompt, so nothing can be reused.
- Changing whitespace or tool order between calls, silently breaking cache hits.
- Relying on prompt instructions alone for JSON instead of using constraints.
- Spreading one conversation across random instances and losing its cache.
- Judging the engine with single, unrelated prompts where prefix reuse cannot help.
- Running long contexts without enough memory for the cache tree.

## Frequently asked questions

### What does SGLang stand for?

The name comes from "structured generation language", reflecting its original focus on programs that structure several model calls and constrain outputs.

### Is SGLang only useful for agents?

No. It is a general-purpose serving engine. Agents, chat with long histories and few-shot pipelines benefit most from prefix reuse, but ordinary single-call workloads also run efficiently.

### Do I have to use the SGLang frontend language?

No. Most users run the server and call its OpenAI-compatible API from any client. The frontend is optional for those who want to express multi-call programs directly.

### How is RadixAttention different from prefix caching in other engines?

The aim is the same: avoid recomputing shared prefixes. RadixAttention organises the cache as a tree that matches any shared prefix automatically and combines it with cache-aware scheduling. Other engines implement prefix caching in different ways, and results vary by workload.

### Can SGLang serve very large models?

Yes. It supports tensor, pipeline and expert parallelism across GPUs and nodes, and disaggregated prefill and decode for large deployments.

## Glossary

- Prefix: the shared beginning of several prompts.
- Eviction: removing cached data to free memory, usually the least recently used first.
- Time to first token: the delay before output begins.
- Few-shot prompt: a prompt containing worked examples.
- JSON schema: a description of the structure valid JSON must follow.
- Mixture of experts: a model that activates only some sub-networks per token.
- Instance: one running copy of the engine with its own GPUs.

## Summary

SGLang is an open-source LLM serving framework whose RadixAttention keeps the KV cache of past requests in a prefix tree and reuses it automatically, which makes multi-turn chat, agents and few-shot pipelines much cheaper. Introduced by LMSYS researchers in late 2023 and published at NeurIPS 2024, it has grown into a general high-performance engine with fast constrained decoding, large-scale parallelism and a prefix-aware router. It works best when prompts are designed to share stable beginnings, and it is worth benchmarking against other engines on real traffic.