Reading documents you can trust: the numbered findings · part 4 of 4
Architecture and the harness
A workflow, not an agent — and policy in code, not in prompts
The question
What did 18 measured findings establish about architecture and the harness?
Our loop is a workflow, not an agent — and that's a feature..
22. Our loop is a workflow, not an agent — and that's a feature.
The model fills answers inside control flow WE wrote; it never chooses the next step. Frameworks add ceremony for this case (agents_2026_style.md, 6 implementations compared).
23. Policy-as-code beats policy-as-prompt — measured.
openai-agents given the verification policy as a prompt: skipped it entirely, 0 tool calls. claude-agent-sdk followed it — at $0.46/question. The synthesis (05_pi_harness): the HARNESS owns the cross-check gate in code; the model is only called for reading and arbitration. An agent that "may verify" when confident, won't.
24. Production routing is sub-task decomposition + a static type→handler table + an effort ladder
(MinerU v3.4.4 source read, 08-04): CV supplies geometry, VLM supplies content, 24-entry lookup table, two effort rungs. That lookup table is the honest baseline any learned router must beat.
25. Freeze channel A across comparative runs.
Re-roll variance on hard questions ≈ ±0.5–1pt at n=300 — reuse baseline_{model}.json or variance confounds every comparison.
26. Which-fields-to-ask is a routed action too
(SchemaRAG lineage, 08-04): retrieving the relevant schema slice improves quality AND cost (+8.8 F1, −48% tokens on large schemas); gain scales with schema size.
29. Most residual DocVQA loss is wording, not seeing
(08-06 failure taxonomy, case_docvqa_failures.md). Of 11 zeros in the best 300-Q run, 6 were the right region with the wrong span — appended count nouns, over-answers, a definition where a term was asked, and one case of the model COMPUTING a total (80) that appears nowhere on the page instead of quoting the printed breakdown. Only 4 were genuine wrong-region errors and 1 was a bad gold label.
30. A shortening pass with a containment gate is the first free lunch we found.
snap may only return a string strictly shorter than AND contained in the frozen draft — it can trim, never rewrite. Result: 6 gains, 0 losses, 0.9536 → 0.9679. Every prior arbiter/editor stage traded fixes for breaks; this one cannot break anything by construction. Same router-not-editor discipline as #16, applied to formatting instead of content.
31. Deterministic string rules beat model calls per dollar.
Three of the six gains came from _clean() with no API call: strip a trailing colon, strip a leading minus from a bare number, and drop a count noun after a number only when that noun already appears in the question ("8 copies" → "8", but "2100 kcal" untouched). Measure the rule's firing set on the full eval before shipping it — this one fires on 2 of 300, both failures.
32. A single stored high-temperature run is not evidence a route works.
Routing localization questions to vote was justified by vote scoring 1.0 on a question the table channel missed; the live rerun scored 0 there with 0.8 agreement. The stored score was a lucky roll. Validate stochastic routes across reruns, not one artifact.
33. A bare mean cannot separate agents — 5 of our 8 were noise
(arena, 08-06). Under a paired bootstrap vs the frozen baseline, only snap ships; table, vote, and both grounded variants have CIs spanning zero. The table channel would need ~16,470 questions to prove its +0.001. Report the required-n with every inconclusive result: "unresolvable at any affordable n" is a decision, "+0.001" is a mirage. Pair the test — every agent answers the same questions, so pairing cancels question difficulty, the dominant variance term.
34. The swarm cannot beat its best member by voting
(arena policy, 08-06). Majority-of-9 = 0.9533 vs snap alone 0.9679; every gated variant also loses. Errors are correlated (L5), so a vote concentrates the shared mistake, and the crowd's verbosity overwrites the best agent's terse formatting. Check the oracle ceiling before building any router: ours is +0.0145 over the best member while the anti-oracle is -0.2381 — 16x risk against — and 42% of remaining loss is in questions no agent answers at all.
35. Two gate designs make a stage unable to regress.
Containment gate (snap): accept only if strictly shorter than AND contained in the draft. External-signal gate (region): the override must be text OCR also found on the page, so a hallucinated heading is discarded. Self-critique gates nothing (L6). Every stage that survived here has an external check in harness code, not in a prompt (L23).
36. Screen new agents on the contested slice; test narrow agents on a trigger pool.
196 of 300 questions are answered perfectly by every prior agent — changing one can only lose — so the screen is 104 contested + 40 controls (48% of a run), validated on snap (all 6 gains inside it). For an agent that fires on 5% of questions, scan a large pool and keep only matches: scanning is free, answering is not (2,500 scanned → 164 triggering, 6.6%).
37. Hold out the questions you designed the agent from.
Prompt rules written after reading a failure are fitted to it; scoring on those questions measures memorisation. arena verdict --holdout reports unseen questions only.
38. A rule written from N failures will fix exactly those N failures.
region (spatial-term rules, written from 2 questions) scored +0.67 on the design set by fixing precisely those 2; on 148 unseen questions of the same type it scored -0.0003, 0 gains, 1 loss. The "TERM: definition" split rule fired once on the design set (+1.00 pt) and zero times on 148 fresh questions. Design-set gains are not evidence. Hold out, or you are measuring your own memory.
39. Verify a weak slice is weak before building for it.
Spatial questions scored 0.8711 on the 300-set, which is what motivated region; 148 fresh spatial questions score 0.9826 on the plain baseline, above its overall average. The gap was small-sample noise around two hard questions. Slice weakness at n=16 is a hypothesis, not a finding.
40. Plausible deterministic rules are often net-negative — measure the firing set.
Stripping a leading "the/a/an" sounds obviously right for DocVQA and is harmful on both sets: gold routinely keeps the article ("THE NUTRITION FOUNDATION, INC.", "the Board of Trustees"). Deterministic rules are free to test against any stored run, so there is no excuse for shipping one unmeasured.
41. Know your detection floor before planning more work.
At n=300 a real 6-question fix carries a ±3.78-point band. Every remaining failure class (over-answer 1.97, reasoning 2.00, perception 1.50) sits below it — the largest would need ~1,100 questions to prove. Of 7.63 points remaining, 2.16 are unreachable (bad gold, punctuation, under-answers) and the rest is four unrelated mechanisms with 1-5 questions each. The bottleneck stopped being ideas and became statistical power; the next investment is ~1,000-1,500 eval questions, not a tenth agent.