AI Insights: Latency & Cost Optimization · part 17 of 17
Robustness run and the regular-vs-hybrid verdict
10 data variants through the rule-engine pipeline, judged on the production rubric
The question
Does the rule-engine pipeline hold production quality across varied customer data — and does hybrid mode earn its extra cost?
8.5/10 average with zero ungrounded numbers in 10/10 scenarios — matches the production baseline at ~40% lower cost. Ship regular mode.
Setup
test-runner.js pushed 10 customer-data variants through the pipeline, scored by a GPT-5.1 judge with the same 10-criteria rubric used for the production baseline (from the speed-ablation phase), so scores are directly comparable to the 8.5+ production bar. Note the scale difference: this rubric is the GPT-5.1 judge scale, not the stricter Claude scale of sub-experiments 8–13.
The run drove two rule fixes
- DeductionChange rule added — a switch from itemized to standard deduction was invisible to the original five rules.
- StateTaxChange got its own cap bucket — a headline state-tax event was being crowded out by the 2-per-Income-category cap.
Regular vs hybrid on the same 10 scenarios
| Mode | Judge avg | Observation added | Cost/call | Latency |
|---|---|---|---|---|
| Regular (rules only) | 8.46 | — | ~$0.0062 | ~5 s |
| Hybrid (+summary) | 8.51 | 1–2 of 10 scenarios | ~$0.0092 | ~6 s |
A statistical tie. Hybrid added an observation in only 1–2 of 10 scenarios — the rules already cover most patterns since the state-bucket fix. Recommendation: ship regular (same quality, 31% cheaper, ~1 s faster); consider hybrid only when the draft count is ≤ 2 — thin returns, where diversity is weakest and an observation adds the most.