Research Threads: What Is Being Measured, and What It Has Cost
Kansas City · Independent · revised continuously
Abstract
Each thread below is a question I have held open long enough to measure. Every one reports its metric with the sample it was measured on and what the measuring cost, and every number cites the file on disk it was read from. Threads that turned out to be dead ends keep their negative results, because a finding that something does not help is the cheapest kind of knowledge to lose and the most expensive to rediscover. Each title below opens the full write-up: the question, what I tried, what the numbers were, and which attempts were wasted. Papers are listed in §2.P; the notebooks behind the numbers are in §3.
2.1Grounded extraction with citations
Eight days, about three and a half dollars of compute, and forty-one numbered findings about reading documents well enough that a person can skip most of the checking. [active · 6 August 2026]
| measure | value | conditions | source |
|---|---|---|---|
| Best answering stack | 0.9679 | ANLS, DocVQA-val, 300-question seeded sample | best_score/, ade/runs/ |
| Fields filled without a human | 84% | and those are 94.1% exact | experiments/RESULTS.md |
| Agents indistinguishable from one plain call | 5 of 9 | paired bootstrap over the arena | ade/SWARM.md |
| Total research spend | $3.50 | of a $9 OpenRouter budget, plus free local CPU | experiments/RESULTS.md |
| Numbered findings | 41 | L1–L41, each citing its run | experiments/LEARNINGS.md |
2.2Inference economics
A panel that took fourteen seconds to appear now takes six and costs about an eighth as much, and the model that does it is smaller and open. Seventeen sub-experiments, about forty dollars. [active · 5 August 2026]
| measure | value | conditions | source |
|---|---|---|---|
| End-to-end answer | 13.9 s → 6.1 s | median of 10 runs | cost-lab/final_experiment |
| First token | 1.5 s → 0.6 s | — | cost-lab/block_tps |
| Cost per answer | −87% | self-hosted 30B open model replacing a hosted frontier model | cost-lab/analysis |
| Quality | held | two independent judges agreeing | cost-lab/llm_as_judge |
2.3Schema-constrained generation
A filing with 369 fields that a language model has to help produce and must never be allowed to write. [paused · 2026-08]
| measure | value | conditions | source |
|---|---|---|---|
| Fields in the target file | 369 | — | wsz-schema-rag/forms |
| Rule the design turns on | the model never emits the file | it proposes values; a deterministic writer emits them | wsz-schema-rag/arch.md |
2.4What agents should remember
Reading memory systems until the parts are ordinary, then building one small enough to run on a laptop, and measuring what recall is actually worth. [active · 2026-09]
| measure | value | conditions | source |
|---|---|---|---|
| Explainer chapters | 11 | written to be read in order | apps/memory/site |
2.5Answers with the slide attached
Ask a question of an hour-long talk and get the answer with the slide it came from, at the timestamp. Then do the same by voice, fast enough that it feels like a conversation. [exploring · 2026-08]
| measure | value | conditions | source |
|---|---|---|---|
| Retrieval on a golden set | 0.917 | recall@10, 14 questions; judge 4.33 / 5; declined 2 of 2 unanswerable | wsz-research/research.md §2 |
| Ingest a 37-minute talk | 52 s | 42.6× realtime on a laptop, $0.042 end to end | wsz-research/research.md §2 |
| Speech recognition on a spot L4 | 417× realtime | $0.0005 per video-hour | wsz-research/research.md §2 |
| Caption cost per keyframe | $0.00024 | an 8B open vision model, measured | video-audio-frames-summarization/research.md §4 |
| Answer latency | 4.5 s | p95 | wsz-research/research.md §2 |
2.6A database that lives in a bucket
HanuDB serves from local disk and keeps nothing that matters there. Delete the machine and a new one rebuilds from the bucket alone. An agent tuned it over 108 experiments while I was asleep. [active · 16 September 2026]
| measure | value | conditions | source |
|---|---|---|---|
| Benchmark score | 9 047 → 40 116 | √(writes/s × reads/s), on a 1-core, 768 MB harness | results.tsv |
| Writes / reads per second | 34 821 / 46 218 | — | run.log of experiment 0091 |
| Experiments | 108 | 74 kept, 27 discarded, 3 crashed | results.tsv |
| Soak | 14 M documents | ten minutes, zero errors | soak-server.log |
| Rebuild from the bucket | 40 s | 4.5 GB snapshot plus tail | README.md, Scale |
| Cost to run | ≈ $0.01 / day | measured on a real free-tier machine | README.md, Real e2-micro |
Read the full write-up — 4 sections → · the preprint · the notebook · source on GitHub (MIT)
2.7Public data, kept honest
The Labor Department's visa disclosure files are wrong in specific, repeatable ways. A loop fixes one of those a night and has to prove it did not move the headline numbers for the wrong reason. [paused · 13 September 2026]
| measure | value | conditions | source |
|---|---|---|---|
| Logged runs | 23 | one change each, headline counts recorded before and after | h1b_firms_list_kansas_city/results.tsv |
| Guardrail | ±3% | metro filings may not move further than this unexplained | results.tsv, kc_filings_last3 column |
| Raw rows to distinct cases | 9.64 M → 7.17 M | four legacy schemas mapped, back to 2010 | results.tsv, run 17 |
| Unresolved locations | 4,056 → 397 | legacy county delineations for one state | results.tsv, run 1 |
| Metro coverage | 5,912 filings | last three years, 1,535 employers | results.tsv, run 19 |
2.PPapers
- Madhu. HanuDB: A Cloud-Durable Document Database for Tiny Compute. Preprint, 2026-09.
- In preparation: grounded extraction with per-field trust verdicts.
- In preparation: what a cheap open model costs to serve, measured end to end.
2.NNotebooks
AI Insights: Latency & Cost Optimization (17 parts); HanuDB, tuned while I slept (6 parts); Reading documents you can trust: the numbered findings (4 parts); Public data, kept honest (4 parts). All notebooks →
Every number in this document is read from a file in the thread's own folder. If a figure here disagrees with its source, the source is right.