Skip to the document
Madhuopen lab

§3 Experiments · AI / LLM

active23 July 20266 August 2026 · 17 parts

AI Insights: Latency & Cost Optimization

Cutting the insights panel from a 24 s wait to under a second, at about a tenth of the cost

GPT-5.1Qwen 30BvLLMSGLangPrompt cachingStreamingLLM-as-judgeRAGRule engineAzureFoundryManaged compute
In one breathTwo viable end states after 17 sub-experiments: self-host Qwen 30B with a facts-grounded prompt for roughly a 10x cost cut at the same quality, or move number selection into a deterministic rule engine and use the model only for wording, so numbers can't be made up. Streaming helps either way and can ship first.

Summary

The AI insights panel in the tax product takes 9 to 24 seconds to show anything. Two reasons: the code waits for the model's complete answer before showing anything, and the model behind it (GPT-5.1, a frontier paid model) is the slowest and most expensive option for this job.

I tested the current inference stack options — vLLM, SGLang, prompt caching, streaming, structured outputs — to find the model and serving setup that does best on accuracy, speed, and cost together. Every option ran the identical production prompt on a synthetic customer return (realistic figures, no real customer data), scored by LLM judges on a fixed 10-point rubric.

Three answers came out of it. Streaming plus a response-length cap fixes the visible complaint today with no new infrastructure. Self-hosted Qwen 30B with a grounded prompt matches GPT-5.1's quality at about a tenth of the cost, with the first word showing up ~18x sooner. And a deterministic rule engine with an AI polish pass removes made-up numbers entirely, at the lowest cost of all — matching the production quality baseline of 8.5/10 while sending the model 50x less data.

Final board — both finalists on the improved prompt, independent Claude judges

Model (single call, V5 facts prompt)Opus 5Sonnet 5First wordTotal timeCost/answer at scale
GPT-5.1 (Azure, paid API)8.07.5~1.6 s~15 s$0.0136
Qwen 30B on vLLM (self-hosted)7.257.0~0.09 s~9 s~$0.0013
Season cost at millions of records, 15 AI calls per return (180M calls)USD per season
GPT-5.1 paid API
$2.46M
Qwen self-hosted (Foundry H100)
~$317K
Qwen self-hosted (A100 PAYG)
~$229K
Rule engine + GPT-5.1 polish
~$1.12M
at 15 calls; $75K at 1 call/return
Time to first word (what the blank screen feels like)milliseconds
GPT-5.1 (Azure)
~1,500 ms
Qwen on vLLM (warm cache)
~114 ms
Qwen on SGLang (warm cache)
~146 ms
GPT-5.6-Luna (reasoning)
~18,100 ms

The tradeoff triangle

Every configuration tested here is a position inside the same triangle: cost, accuracy, and performance pull against each other, and improving one usually gives up another. The 17 sub-experiments measure where each model and serving setup lands on all three corners at once — the finalists are the ones that hold the accuracy corner while pulling cost and performance in at the same time.

Tradeoff triangle for LLM deployments with three corners connected by cost, accuracy, and performance edges
The tradeoff triangle for LLM deployments — every serving decision in this experiment is a move between cost, accuracy, and performance.

The protocol, in order

01Baseline: the production call as it runs todayThe 9–24 s blank screen comes from waiting for the full answer: the call is blocking, non-streaming, and has no length cap. It's not a frontend problem.insight02Cheaper paid models: can a smaller GPT do the job?Nothing cheaper on the paid side matches GPT-5.1. The mini/nano reasoning tiers add cost and delay on a short structured task.rejected03Parallel calls: split the work into fourFaster on the clock, but asking a call to produce 'credit insights' makes it invent ineligible credits. A single call keeps the model honest.rejected04Open models, raw: Qwen vs Llama vs DeepSeekNo open model matches GPT-5.1 out of the box — but the failures look like missing information, not missing capability. Qwen is the one worth fixing.insight05Grounding the prompt: feed the model the factsQwen jumped 5 → 8/10, level with GPT-5.1. The gap was facts and rules, not model weights.adopted06Serving engines: vLLM vs SGLang, and the prefix cacheEither engine works. The big win is the prompt cache itself: warm first word ~114 ms, about 18x faster — it's what makes self-hosting cheap.insight07Throughput and the season cost modelSelf-hosting beats the API from ~270 answers/hour, and measured throughput runs about 10x past that break-even. The gap is large, not marginal.insight08An independent judge: does the quality story survive?The ranking survives, absolute scores drop 1.3–1.5 points (the old scores were inflated), and the independent judges caught a fabrication the self-judge missed.insight09Closing the gap: rules don't work, facts doFive verified reference facts alone lifted Qwen a full point to near-parity. Behavioral rules did nothing. Stacking both made it worse.adopted10GPT-5.1 speed ablation: what actually makes the API fasterOutput length is the only real lever: a terse prompt cut wall time 32% and cost 33% with quality held at 8.0.adopted11Foundry managed compute: Qwen without running the VMManaged H100 is ~15% costlier per answer than the DIY A100 but 1.6x faster wall time — a workable option that removes VM ops.insight12Improvement ablation W0–W9: single tweaks vs combosNo single tweak moved the needle; combos did — facts2+fewshot (W7) and combo+code-gate (W8) narrowed the gap to GPT-5.1 to ~0.35.insight13Final validation: the W8 recipe at n=10Parity reached: W8 Qwen scores 7.95 vs GPT-5.1's 8.0 (Opus), zero ungrounded numbers, 6.1 s wall — 87% season cost cut.adopted14GPT-5.6-Luna: tested and disqualifiedBest-in-test quality (8.0/7.75), but 1,926 hidden reasoning tokens push TTFT to 18 s and real cost 30% above GPT-5.1.rejected15Rule-engine POC: compute the numbers, let AI only writeNumbers, categories, and priorities come from code, so they can't be made up — 4.0–4.6 s, ~$0.0062/call, and the model never sees the raw ORM.adopted16Hybrid mode: short facts summary vs the raw ORMA short 3-year facts summary works: the model found the income trend in 2/2 runs. Feeding it the full raw ORM found nothing in 2/2 runs at 5x the tokens. Hybrid's value on thin returns still needs data.needs more data17Robustness run and the regular-vs-hybrid verdict8.5/10 average with zero ungrounded numbers in 10/10 scenarios — matches the production baseline at ~40% lower cost. Ship regular mode.adopted