AI Insights: Latency & Cost Optimization · part 3 of 17
Parallel calls: split the work into four
One call per category (income, deductions, credits, life events) running in parallel
The question
Can four smaller simultaneous calls — one per insight category — beat one big call on latency without losing quality?
Faster on the clock, but asking a call to produce 'credit insights' makes it invent ineligible credits. A single call keeps the model honest.
What happened
The four calls overlap, so wall time drops. But when a call is asked to produce "credit insights" on its own, it produces one even for a client who qualifies for no credits — it invents an ineligible one to fill the slot. On the test client the parallel mode wrongly pushed the Saver's Credit in 9 runs out of 10, and Qwen's quality score fell from 8 to 5.5. GPT-5.1 resisted the trap (8.5 parallel); the cheaper models didn't.
It also costs more: the shared ~2,600-token instruction block is sent four times instead of once.
A later check
Re-running with a realistic customer (real $135k filer instead of a sparse $0-income test case) confirmed it from the other side: single beats parallel for high-income filers too — parallel invents credits there as well. The sparse test case had a manufactured EITC bug that made parallel look worse than it was, but the conclusion held with representative data.