AI Insights: Latency & Cost Optimization · part 1 of 17
Baseline: the production call as it runs today
Exact production prompt on GPT-5.1, measured end to end
The question
What is the starting point — quality, speed, and cost of the insights call as production runs it?
The 9–24 s blank screen comes from waiting for the full answer: the call is blocking, non-streaming, and has no length cap. It's not a frontend problem.
Setup
A clean reproduction of the production call: system prompt is the verbatim EnhancedInsightsPromptTaxBracketSingleYear.txt; user prompt is the output of BuildSingleYearPrompt() for a synthetic customer return (single filer, $135k wages, owes ~$21k), taken from the repo-root orm.json and current_ai_response.json; response_format=json_object to match production. No hardening, no RAG, no citations — the point is to measure what runs today.
Every request sends roughly 2,600 tokens of input — most of it the fixed instruction block, with a small tail of client-specific numbers. The model writes back 600 to 1,300 tokens, about 6–8 insights per answer. Production makes this as one blocking, non-streaming call (CallEnhancedOpenAIServiceAsync / CallClientOpenAIServiceAsync). That wait is the 8–18 s latency users complain about.
Results (median of runs, GPT-5.1 as judge)
| Mode | Quality | Total time | Cost/answer | Ungrounded numbers |
|---|---|---|---|---|
| Single call (production shape) | 8.7 | 21–24 s | $0.0136 | 0 |
| 4-way parallel by category | 8.5 | ~8 s | $0.020 | 0 |
What the code inspection found
- The call is blocking and non-streaming — the screen waits for the full answer before showing anything.
- There is no cap on answer length — output tokens are priced 8x input, so long answers dominate the bill.
- The prompt is ~2,600 tokens and almost all of it is identical across calls — a good fit for prompt caching (used later in sub-experiment 6).