Serving engines: vLLM vs SGLang, and the prefix cache

Same Qwen model on both engines, plus a 10-customer cold/warm cache test

The question

With Qwen chosen, which serving engine — and how much does prompt caching actually buy on real traffic patterns?

insight

Either engine works. The big win is the prompt cache itself: warm first word ~114 ms, about 18x faster — it's what makes self-hosting cheap.

vLLM warm TTFT~114 mssteady 112–123 ms

SGLang warm TTFT~146 msone 7 s warmup blip

Cache speedup18.5x / 6.6xvLLM / SGLang cold→warm

Single-stream wall7.6 vs 9.0 sSGLang vs vLLM

Why caching works here

The prompt is almost entirely identical on every call: the ~2,600-token instruction block (role, rules, tax facts) never changes; only the client's numbers do, and those are a small tail at the end. The engine computes the shared block once, caches it, and reuses it — so nearly every production call is "warm" and gets its first word almost instantly. This cache is the single biggest reason self-hosting is cheap for this workload: the long prompt's prefill is paid once, not on every one of the millions of calls.

Cold/warm test: same instruction block, 10 different customers

Cold/warm test: same instruction block, 10 different customers
Call	Customer	SGLang first word	vLLM first word
1 (cold — fills cache)	Single $135k	968 ms	2,112 ms
2	Single $62k	7,055 ms *	114 ms
3	MFJ $210k	141 ms	115 ms
4	HoH $88k	144 ms	112 ms
5	Single $320k	241 ms	113 ms
6	MFJ $96k	143 ms	112 ms
7	Single $45k	143 ms	115 ms
8	MFJ $150k	147 ms	113 ms
9	HoH $175k	151 ms	118 ms
10	Single $410k	146 ms	123 ms

* SGLang's call 2 spiked to 7 s — a one-off warmup/compile blip; calls 3–10 are its true steady state (~145 ms).

Engine head-to-head (same Qwen 30B, quality 8/10 on both)

Engine head-to-head (same Qwen 30B, quality 8/10 on both)
Engine	Precision	Wall (single)	TTFT	Cost/answer	Throughput @conc1	@conc16
SGLang	BF16 (FP8 blocked on A100)	7.6 s	131 ms	$0.0077	540 req/hr	2,880 req/hr
vLLM	FP8 (Marlin kernels)	9.0 s	92 ms	$0.0091	420 req/hr	2,880 req/hr
GPT-5.1 (reference)	—	24 s	1,475 ms	$0.0136	—	—

vLLMSGLangRadixAttentionprefix cachingTTFT