§3 Experiments · AI / LLM
AI Insights: Latency & Cost Optimization
Cutting the insights panel from a 24 s wait to under a second, at about a tenth of the cost
Summary
The AI insights panel in the tax product takes 9 to 24 seconds to show anything. Two reasons: the code waits for the model's complete answer before showing anything, and the model behind it (GPT-5.1, a frontier paid model) is the slowest and most expensive option for this job.
I tested the current inference stack options — vLLM, SGLang, prompt caching, streaming, structured outputs — to find the model and serving setup that does best on accuracy, speed, and cost together. Every option ran the identical production prompt on a synthetic customer return (realistic figures, no real customer data), scored by LLM judges on a fixed 10-point rubric.
Three answers came out of it. Streaming plus a response-length cap fixes the visible complaint today with no new infrastructure. Self-hosted Qwen 30B with a grounded prompt matches GPT-5.1's quality at about a tenth of the cost, with the first word showing up ~18x sooner. And a deterministic rule engine with an AI polish pass removes made-up numbers entirely, at the lowest cost of all — matching the production quality baseline of 8.5/10 while sending the model 50x less data.
Final board — both finalists on the improved prompt, independent Claude judges
| Model (single call, V5 facts prompt) | Opus 5 | Sonnet 5 | First word | Total time | Cost/answer at scale |
|---|---|---|---|---|---|
| GPT-5.1 (Azure, paid API) | 8.0 | 7.5 | ~1.6 s | ~15 s | $0.0136 |
| Qwen 30B on vLLM (self-hosted) | 7.25 | 7.0 | ~0.09 s | ~9 s | ~$0.0013 |
The tradeoff triangle
Every configuration tested here is a position inside the same triangle: cost, accuracy, and performance pull against each other, and improving one usually gives up another. The 17 sub-experiments measure where each model and serving setup lands on all three corners at once — the finalists are the ones that hold the accuracy corner while pulling cost and performance in at the same time.
