AI Insights: Latency & Cost Optimization · part 15 of 17
Rule-engine POC: compute the numbers, let AI only write
Deterministic draft generation (~10 ms) + GPT-5.1 polish-only call
The question
What if insight selection isn't an AI problem at all — can a deterministic rule engine draft the insights and the model only polish wording?
Numbers, categories, and priorities come from code, so they can't be made up — 4.0–4.6 s, ~$0.0062/call, and the model never sees the raw ORM.
How it works
The rule engine walks the year-over-year comparison table from the real data feed (data/client-get.json + data/brackets.json — note TaxBracketInfo.val is stringified JSON), applies 5 rules with thresholds, ranks with a tiered scheme, caps at 2 per category / 4 total, and emits draft.json — all deterministic, in ~10 ms, no dependencies.
GPT-5.1 then gets only the drafts — never the raw ~59 KB ORM — and rewrites the wording into the production insight envelope (insights.json). The polish prompt is ~1,034 tokens. Every number, category, and priority was computed in code, so those fields can't be made up; the model can only rephrase.
Season economics (millions of records)
| Calls per return | Rule engine + polish (regular) | Hybrid mode | Prior full-prompt baseline |
|---|---|---|---|
| 1 | $75K | $111K | $179K |
| 5 | $375K | $553K | $894K |
| 10 | $749K | $1.11M | $1.79M |
| 15 | $1.12M | $1.66M | $2.68M |
At Azure GPT-5.1 rates ($1.25/$10 per 1M): regular mode ~$0.0062/call (1,034 prompt / 495 completion tokens, ~5 s) vs the prior full-prompt baseline's $0.0149. See costs.md for the derivation.