AI Insights: Latency & Cost Optimization · part 13 of 17
Final validation: the W8 recipe at n=10
Combo prompt + 30-line post-code on Foundry H100, full throughput sweep
The question
Does the W8 recipe hold at n=10 with a full throughput sweep — enough to call it the ship candidate?
Parity reached: W8 Qwen scores 7.95 vs GPT-5.1's 8.0 (Opus), zero ungrounded numbers, 6.1 s wall — 87% season cost cut.
The recipe
W8 = single call + combo prompt (V5 facts + facts2 + few-shot) + ~30 lines of post-processing code (eligibility gate + insight merge) + streaming. W7 (same without the post-code) scored 7.5/7.0 and serves as the fallback if the post-processing layer can't ship.
Results (n=10, Foundry H100 BF16)
| Metric | Qwen W8 | GPT-5.1 + V5 | W7 fallback |
|---|---|---|---|
| Opus 5 judge | 7.95 | 8.0 | 7.5 |
| Sonnet 5 judge | 7.0 | 7.5 | 7.0 |
| Ungrounded numbers | 0 | 0 | — |
| Wall time | 6.1 s | ~15 s | — |
| TTFT | 614 ms | ~1.6 s | — |
Throughput sweep (combo prompt)
3,960 req/hr at concurrency 16 → $0.00176 per answer. At millions of records x 15 views, season cost ~$317K vs ~$2.46M on the paid API — an 87% cut with quality at parity.