§3 Experiments · product
paused12 September 2026 → 13 September 2026 · 4 parts
Public data, kept honest
An autonomous loop that makes the Department of Labor's H-1B disclosure files more accurate one fix at a time — and has to prove each fix did not move the numbers for the wrong reason.
In one breath22 runs, 21 kept. Every LCA filing since fiscal 2010, resolved to a metro through ZIP, county and CBSA; employers clustered by tax ID. The dataset behind bigImmigrationHub.
Summary
Disclosure files are messy in specific, repeatable ways: the same city spelled three ways, one employer under several tax identifiers, a legacy schema that reads dates differently every few years. The loop attacks one at a time. Its guardrail is the headline itself: if Kansas City's filing count moves by more than three per cent, the run must explain exactly why or it is discarded.
The protocol, in order
01Geography: where the job actually isHistory fixes: date parsing (mixed date/datetime strings dropped FY2013/14/19), population-weighted ZCTA->county (Census 2010 rel file; fixes Boise/Bartlesville), city fallback share 0.8->0.6 — all 17 FYs now populated (national…adopted02Employer identity: one company, one rowEmployer fields: display name prefers longer spelling when top spelling is a truncation; wage P25/P75, prevailing-wage level counts, top SOC codes (last 3 yrs) — no count changes; runtime 281s (watch the 5-min budget)adopted03History and coverage: every file since 2010History: DOL LCA files FY2010-FY2022 (4 legacy schemas mapped in scripts/convert_lca_legacy.py) + USCIS hub FY2010-FY2021 via Chrome agent; fiscal_years now 2010-2026; ZIP zero-padding — 9.64M rows -> 7.17M cases; KC last-3 headline…adopted04Deliverables: what people asked forDeliverable (teammate session request): KC Indian-origin population by citizenship from ACS 2019-23 PUMS + SEVIS India share -> data/out/kc_indian_breakdown.json — KC Asian Indian alone 19,104 +-2,079: US-born 6,958 / naturalized 5,438 /…insight