AI Insights: Latency & Cost Optimization · part 4 of 17
Open models, raw: Qwen vs Llama vs DeepSeek
Three self-hosted models on the unmodified production prompt, vLLM on Azure A100
The question
Can a self-hosted open model match GPT-5.1's quality on the exact production prompt, with no modifications?
No open model matches GPT-5.1 out of the box — but the failures look like missing information, not missing capability. Qwen is the one worth fixing.
Results (median of 3 runs, GPT-5.1 judge)
| Model | Single call | Parallel | Total time | Cost/answer | Failure mode |
|---|---|---|---|---|---|
| Qwen3-30B-A3B-FP8 | 5 / 10 | 5 / 10 | 5–5.7 s | $0.005–0.006 | under-generates (2–3 insights); stale limits |
| Llama 3.3-70B-FP8 | 6 / 10 | 5 / 10 | 16.6–23 s | $0.017–0.023 | slow, barely fits A100, no quality win |
| DeepSeek R1-Distill-Qwen-32B | 6 / 10 | — | 15–19 s | $0.015–0.020 | reasoning overhead; ineligible credits |
| GPT-5.1 (reference) | 8.7 | 8.5 | 8–21 s | $0.009–0.018 | — |
The shared failure pattern
Every open model failed the same way: stale contribution numbers (old 401k limits instead of the current year's), invented ineligible credits, and too few insights per single call. The failures didn't depend on the model — which pointed at missing information, not missing capability. Bigger (Llama 70B) and reasoning-style (DeepSeek) models didn't fix it; they just cost more.
Qwen's mixture-of-experts design activates only a small slice of parameters per token, and the FP8 quantization fits comfortably on one A100 — making it by far the fastest and cheapest to serve (TTFT ~115 ms).