HanuDB, tuned while I slept · part 4 of 6
adopted14 September 2026cost: local CPU; a free-tier VM for the real runs
Recovery: snapshots, rebuilds and the bucket
Proportional to the log tail, not to the data
The question
Of 16 attempts in this theme, which ones survived two benchmark runs, the correctness suite and the gates?
adopted
Process discipline: a set -e bug in keep.sh (empty experiments/ made ls | wc fail) meant the baseline snapshot never advanced; a later discard reset the tree to day-one code and three experiments measured the wrong program. The scripts now verify…
Attempts16
Kept9
Discarded or crashed0
Best kept score39,656
Every run in this theme
| # | score | writes/s | reads/s | verdict | what changed |
|---|---|---|---|---|---|
| 0004 | 8,895.2 | 8,439 | 9,376 | keep | recovery: seq/prev_hash from local seg files (meta.json lags); handle raw file; correctness keep (score = noise) |
| 0014 | 13,623.2 | 24,908 | 7,451 | keep | RECONSTRUCTION: keep.sh had a set -e bug (empty experiments/ dir) so .baseline never advanced and the discard at 0010 reset src to 0001; all kept patches (0002 indexer… |
| 0039b | 6,230.8 | 3,360 | 11,554 | info | REAL e2-micro in twin mode (us-central1 + us-east1 buckets): writes still fsync-bound (p50 8.1 ms), rebuild from the two real buckets 33637 docs in 8.2 s, 0 mismatches;… |
| 0041b | — | 2,822 | — | info | REAL e2-micro: 43 MB physical snapshot packed in 4.3 s, multipart-uploaded to both regions; rebuild from GCS = extract 0.66 s + replay 23.5k records = 4.1 s total (was… |
| 0056b | — | — | — | info | REAL e2-micro on the latest build: verify OK (38 segments, 1 snapshot), dump from the VM streamed all 196759 documents from the real buckets |
| 0063 | — | — | — | keep | TEST: scripts/smoke.sh regression suite (41 checks: auth, CRUD, filters, text, sort, aggs, idempotency, min_txn, SIGKILL restart, rebuild, twin loss, snapshots, compact,… |
| 0065 | 32,930.9 | 28,023 | 38,699 | keep | snapshot download streamed to a file and extracted from it (a 741 MB snapshot held in memory OOM-killed the 768 MB container). SCALE TEST: 2.45M docs written in 120 s… |
| 0065b | — | 1,025 | 89 | info | REAL e2-micro SOAK 180 s: writes collapse to 1025/s, reads 89/s (p99 1.7 s), 22.5k writes not yet in the bucket at crash time, rebuild 95 s. Hypothesis: e2-micro… |
| 0075 | 34,159.6 | 27,617 | 42,252 | keep | SAFETY: snapshot extraction streams entry by entry with fdatasync every 32 MB (rebuild of a 2.7 GB snapshot OOM-killed the cgroup). Bench unchanged |
| 0082 | 34,390.1 | 28,475 | 41,533 | keep | LATENCY: the snapshot pack runs on its own thread at nice 10 (setpriority on the thread; dies with it) so requests and the indexer win the single core during a 30 s… |
| 0085 | 32,228.7 | 28,047 | 37,215 | keep | OBSERVABILITY+MEMORY: io log now has wb_mb (fjall write buffer), sealed memtables and index_lag; malloc_trim after recovery (anon 294 -> 238 MB after the bench rebuild:… |
| 0091a | — | — | — | info | MIGRATION TEST for 0091: a 0088 binary wrote 3000 docs (fjall layout, bucket snapshot + segments); the 0091 binary opened the dir, migrated 3000 docs to lsm_tree in 58… |
| 0092 | 39,656 | 34,336 | 45,800 | keep | SAFETY (fixes for 0091): meta.next_txn (the current.log staleness floor) follows applied_txn = min(kv_flushed, index) again; local-bucket uploads go to <bucket>/.upload/… |
| 0097a | — | — | — | info | EXPORT ON THE VM: POST /v1/_admin/export?col=bench on hanudb-1 wrote 41922 docs (5.39 MB zstd JSONL) to both real buckets in 2.7 s; DuckDB 1.5.5 read_json(...,… |
| 0104 | 34,871.4 | 34,937 | 34,806 | keep | SAFETY+TOOL: snapshot tar entries are cut at their header size (tar::Builder::append_dir_all copied the whole file: a kv table still being flushed grew meanwhile and… |
| 0104b | — | 20,855 | — | info | UPDATE-HEAVY SOAK (5 min, 50/40/10 insert/update/delete, 768 MB): 6.27M ops at 20.9k/s, 0 errors, anon 220-317 MB flat (tantivy deletes + kv tombstones under control),… |