Reading documents you can trust: the numbered findings · part 2 of 4
What does not work
Measured dead ends — kept so nobody pays for them twice
The question
Which popular techniques did the measurements say to stop using?
Systematic errors can't be outvoted..
5. Systematic errors can't be outvoted.
Self-consistency k=5 @ T=0.7: 0.9513 ≈ baseline 0.9525; 291/300 questions unanimous across 5 samples, including the wrong ones. The model is not a shaky clerk; it misreads the same way every time.
6. A judge that edits answers breaks as much as it fixes.
Clean test with frozen channel A: arbiter overrides fixed 0, broke 2. Verification machinery must GATE and ROUTE answers, never rewrite them ("router, not editor"). Snap-to-candidate (arbiter output fuzzy-matched back to a candidate string, space-stripped ≥90) is mandatory or formatting rewrites eat semantic wins.
7. Weak extra voters add nothing.
Trivote with qwen-8b as third voter: no gain over the strong baseline. A weak challenger also drags the arbiter into overruling correct answers (0.9525 → 0.9428 with weak channel B).
8. Same-model self-verification is noise.
Model self-confidence is nearly worthless (its errors are consistent); the market ships it anyway (Reducto Deep Extract, Extend Review Agent — see trust_layer_survey.md). Literature: LLMs cannot self-correct reasoning without external signals (2310.01798), self-critique unreliable (2505.17656, 2602.06948).
9. Doc-level OCR routing is flat.
Oracle per-doc backend routing = 108/113 = Mistral-OCR-alone on our bench. Hosted generalist OCR already reads neat handwriting and degraded scans; routing pays only for cost/latency, not accuracy.
10. Multi-OCR word voting: skip.
Published gains live on degraded scans only; correction layers over-correct clean text (ICDAR 2026).
11. Markdown-in-prompt poisons answer formatting.
OCR text in the primary prompt makes the model copy OCR artifacts verbatim. The image answer stays primary; OCR text is a cross-check channel only.
12. Over-terse prompts cause abbreviation errors
("USDA" for the printed full name). And parallel API bursts trip per-key surge limits — sequential with ~1.5s pacing.