Grounded extraction with citations

1Resolution before intelligence

The first real finding was embarrassing, in the sense that it cost nothing and I could have found it on day one. Feeding pages to the model at 2400 pixels instead of 1540 was worth +0.7 ANLS. That was the single largest cheap win of the whole project, and it is not a modelling result at all.

One W-2 made it concrete. A 620-pixel scan of it produced zero readable fields, and when I upscaled that same scan the model confidently read two values out of the wrong cells. The 1536-pixel original of the same form was simply correct on the first pass. The model was never the problem. The picture was.

The matching negative result: swapping the reader for a bigger frontier model does not fix this. The open vision model I settled on tops DocVQA-test at 0.971, above the frontier models, at somewhere between four and twenty-four times less cost — and the specific failure that hurts most, reading a value out of the row next to the right one, shows up in every model family that has been measured. Paying twenty-four times more buys the same class of mistake.

2What does not work, which took longer to establish

I spent most of the middle of this project on inference-time machinery — judges, voting, crops, running three readers and combining them. Every one of those schemes landed between 0.94 and 0.95 on DocVQA. The plain reader at 2400 pixels was 0.9525. In other words the entire category was worth nothing, and I had to run it to know that.

The cheerful reading of all this is that the reader's quality is set by the reader, and no amount of scaffolding adds accuracy it does not have. The gap between my 0.9525 and the leaderboard's 0.9725 is protocol and question ambiguity, not something a better pipeline recovers.

3What did work: diversity, not repetition

Accuracy went from 0.9364 to 0.9679 and none of it came from a bigger model. Larger images, never letting the model tell me where on the page something was, reading each document twice by two genuinely different routes rather than twice by the same route, and a final pass that is only permitted to delete words, never to add them.

That last one is the closest thing to a free lunch I found. Most of what remained wrong on DocVQA was wording rather than seeing: the model had the right value and wrapped it in extra words. A shortening pass with a containment gate — the shortened answer must be a substring of the original, or it is rejected — fixes that class without being able to invent anything.

The trust machinery is built from cheap deterministic evidence rather than more model calls. A checksum that fails, a date that cannot exist, a total that does not match its lines, two independent routes disagreeing, a value whose citation box could not be found. A missing citation in particular turned out to be a signal rather than a failure: if the system cannot point at where on the page it read something, that is exactly the value a human should look at.

Null needed splitting into three. Not present on the document, present but unreadable, and not asked for. Collapsing those into one empty value destroys the only information the reader has about why a field is blank.

The result: 84% of fields come back without a human touching them, and those are 94.1% exact. The other 16% are routed to a person with the box drawn on the page.

4The agent arena, and why it needed a statistic

Late in the project I built an arena to compare nine agent designs against one plain model call. The first scoreboard was a table of means and it was useless. A bare mean cannot separate designs whose difference is smaller than the run-to-run noise, and mine were.

Once I ran a paired bootstrap over the arena, five of the nine agents turned out to be statistically indistinguishable from a single plain call. They had all looked like small improvements. They were sampling noise that I had been reading as progress for two days.

The other thing the arena settled: a swarm cannot beat its best member by voting. If one design is genuinely better, averaging it with weaker ones moves the result toward the weaker ones. That is arithmetic, but I did not believe it until the numbers said so.

The loop that runs all of this is a workflow, not an agent — a fixed graph of steps with a static table mapping field type to handler, and an effort ladder that only escalates when the cheap path fails its gate. I keep catching myself wanting to make it agentic. Every time I have measured that instinct, it has cost money and added variance.

5What is still open

The inference-time phase on DocVQA is closed at 0.9525 and I do not think more scaffolding moves it. What is not closed: the trust verdicts are tuned on a seeded 300-question sample, which is small, and the field-level numbers come from my own document set rather than a public benchmark. A held-out set built by somebody else is the obvious next thing and I have not done it.

Finding 38 is the one I keep re-learning: a rule written from N observed failures will fix exactly those N failures. Several plausible deterministic rules turned out net-negative once I measured how often they fired on documents they were not written for.

6Evidence

Every step was written down at the time as a claim, its evidence and a verdict — Reading documents you can trust: the numbered findings (4 parts). The discarded runs are in there too.

Where the work lives:

• ~/company/waz-citation-pdf — the lab and the extraction library.
• ~/company/waz-citation-pdf/experiments/LEARNINGS.md — 41 numbered findings.
• ~/company/waz-citation-pdf/experiments/RESULTS.md — every measured run.
• ~/company/waz-citation-pdf/ade/SWARM.md — the agent arena and why it needed a metric.