AI Insights: Latency & Cost Optimization · part 16 of 17
Hybrid mode: short facts summary vs the raw ORM
Can the model add one grounded observation the rules missed — and what data should it see?
The question
Rules can't anticipate every pattern. Can the model contribute one extra 'AIObservation' — and does it need the raw data or just a summary of it?
A short 3-year facts summary works: the model found the income trend in 2/2 runs. Feeding it the full raw ORM found nothing in 2/2 runs at 5x the tokens. Hybrid's value on thin returns still needs data.
How hybrid works
The polish prompt additionally carries a short summary of the customer's numbers (the SignalDigest in the code) — a 3-year view of all metrics plus the W-4 profile, +865 input tokens. The model may append at most one 'AIObservation' insight, with numbers quoted verbatim from that summary, confidence capped at 75, consultative tone. It worked: the model flagged a 3-year income trend ($80k → $100k) that no threshold rule covered.
ORM-as-input: rejected
The obvious alternative — just send the full raw ORM and let the model find patterns — failed cleanly. The full ORM costs 5,228 prompt tokens and the model added no observation in 2 of 2 runs: a mostly-null 59 KB JSON hides the signal. The short summary surfaced a trend in 2 of 2 runs at a sixth of the tokens. Higher cost, worse output — don't send the ORM. (The --orm mode is kept in call-ai.js for re-verification.)
Cost comparison
| Mode | Prompt / completion tokens | Latency | Cost per call |
|---|---|---|---|
| Regular (rules only) | 1,034 / 495 | ~5 s | ~$0.0062 |
| Hybrid (+summary) | 1,891 / 685 | ~6 s | ~$0.0092 |
| ORM-as-input | 5,228 / — | — | rejected: no observations produced |