AI Insights: Latency & Cost Optimization · part 12 of 17
Improvement ablation W0–W9: single tweaks vs combos
Ten prompt/post-processing variants in one GPU burst, judged offline — plus a RunPod detour that failed
The question
Which further improvements — more facts, few-shot examples, post-processing code, self-refine — actually raise Qwen's score, alone or combined?
No single tweak moved the needle; combos did — facts2+fewshot (W7) and combo+code-gate (W8) narrowed the gap to GPT-5.1 to ~0.35.
The RunPod detour (failed)
RunPod serverless failed twice with container-RAM OOM during CUDA-graph capture, burning ~$5.50 with no logs available via API and no host control — the endpoint and template were deleted. Lesson: serverless GPU platforms hide exactly the knobs this work needs; use pods or Azure.
The burst pattern that replaced it
All 48 generations for 10 variants ran in one 27-minute/$3.20 Azure Foundry H100 burst with auto-teardown, and all judging happened offline afterward (judge_all.py). Separating GPU time from judge time cuts each experiment cycle to ~$1.25. This became the standard pattern for every later run.
Results (10 variants, n=6, Opus / Sonnet)
| Variant | Opus 5 | Sonnet 5 | Note |
|---|---|---|---|
| W0 baseline (V5 prompt, BF16) | 7.1 | 6.25 | reference |
| Single tweaks (each alone) | ≤ +0.15 | — | none significant |
| facts2 alone | 6.6 | — | HURT: model paraphrases new facts into derived numbers |
| W7: combo facts2 + fewshot | 7.65 | 6.8 | best prompt-only variant |
| W8: combo + eligibility gate + merge (code) | 7.55 | 6.75 | ungrounded 0 — production pick |
| W9: combo + self-refine pass | 7.75 | 6.5 | highest Opus but 2x latency |