AI Insights: Latency & Cost Optimization · part 9 of 17
Closing the gap: rules don't work, facts do
Seven prompt variants, one change at a time, dual Claude judges
The question
Can Qwen's judge score be improved by changing the prompt — testing one change at a time?
Five verified reference facts alone lifted Qwen a full point to near-parity. Behavioral rules did nothing. Stacking both made it worse.
Design
Seven variants x 6 runs each, all scored by both Claude judges. Three variants each added one behavioral rule (never invent figures; unique priorities; one insight per tax lever), one stacked all rules, and V5 added five verified reference facts with no rules at all: the SALT cap ($10k), the 7.5% medical-expense AGI floor, the 110% estimated-tax safe harbor, the $1,000 underpayment-penalty threshold, and FSA limits.
Results (Opus 5 / Sonnet 5)
| Qwen prompt variant | Opus 5 | Sonnet 5 | Mechanical metrics |
|---|---|---|---|
| Baseline (grounded prompt from sub-exp 5) | 6.3 | 5.5 | — |
| + each behavioral rule (best of 4 rule variants) | 6.05 | 6.0 | eligibility violations → 0, priority ties → 0 |
| + five reference facts only (V5) | 7.25 | 7.0 | ungrounded 0, no_hallucination 4.5 → 7 |
| + facts AND all rules stacked (V6) | 6.1 | 5.5 | over-constrained, regressed |
Why
Qwen wants to quantify no matter what. When the constants it reaches for are missing from the prompt, it invents them (often wrong); when they're supplied, it cites them. Grounding rules are simply ignored at 30B scale — each rule fixed its mechanical target metric without moving the judge score, because the model kept inventing derived dollar math regardless of being told not to.
The same five-fact prompt was then given to GPT-5.1 (n=10): it improved too, 7.35 → 8.0 / 7.5 with zero unsupported numbers in all ten runs. The facts help every model, not just Qwen. My prediction before the run — that facts were already saturated — was wrong; facts supply was the limiting factor.