HanuDB, tuned while I slept · part 2 of 6
adopted14 September 2026cost: local CPU; a free-tier VM for the real runs
Reads: filters, sort and full-text on one core
Caches that are safe because segments never change
The question
Of 39 attempts in this theme, which ones survived two benchmark runs, the correctness suite and the gates?
adopted
On one core, every background thread competes with request handling: moving work off the write lock onto a spare thread (0003) made throughput worse. Blocking writers briefly is free; extra CPU is not.
Attempts39
Kept21
Discarded or crashed13
Best kept score37,280.7
Every run in this theme
| # | score | writes/s | reads/s | verdict | what changed |
|---|---|---|---|---|---|
| 0002 | 9,237.9 | 9,650 | 8,843 | keep | indexer thread owns IndexWriter (channel, bounded 10k); no RwLock or commit task; +2% (noise) but structural |
| 0007 | 12,204.4 | 25,342 | 5,877 | discard | fjall block cache 64 -> 256 MB: no change in reads (5877 vs 5926/s), +180 MB RSS; the read cost is not cache misses |
| 0008 | 13,496.8 | 25,271 | 7,208 | keep | index paths namespaced by collection (k.{col}.path, t.{col}._all); no col MUST term (was 70% of read CPU in the posting intersection); id term = col+id (cross-collection… |
| 0010 | 13,527.9 | 25,386 | 7,209 | discard | commit interval 500 -> 2000 ms: identical (reads 7209 vs 7208/s); segment count is not what limits reads; keep 500 ms visibility |
| 0011 | 12,026.7 | 24,782 | 5,837 | discard | id as STRING|FAST column instead of doc store: reads -19% (5836 vs 7208/s); a 500k-term dictionary lookup per hit is slower than a doc-store block read |
| 0016 | 13,851.9 | 28,921 | 6,634 | keep | text field holds only the joined _all string per collection (no per-field text copy): writes +16% (28.9k/s), rebuild 579k docs in 10.1 s at 280 MB peak (was 498k in 12.2… |
| 0017 | 14,016.2 | 30,072 | 6,533 | discard | HANUDB_THREADS=1 (single tokio worker): +1.2% (noise); real e2-micro has 2 vCPUs so the default stays |
| 0018 | 16,113.1 | 29,119 | 8,916 | keep | string equality / $in filters evaluated as fast-column predicates in the collector when a text clause drives the query (no posting intersection): reads 6634 -> 8916/s… |
| 0019 | 20,354.5 | 28,932 | 14,320 | keep | equality filters as per-segment cached bitsets (built once from postings, 32 MB budget) applied in a Query/Weight wrapper so tantivy block-max pruning still works; most… |
| 0021 | 22,129.8 | 29,349 | 16,687 | keep | sort by numeric/bool fast column through a per-segment cached Column<u64> (custom SortKeyComputer; strings keep tantivy's erased path): reads 14320 -> 16687/s (+17%),… |
| 0022 | 22,912.8 | 29,123 | 18,027 | keep | fjall docs keyspace: data_block_hash_ratio_policy(0.75) (hash index inside data blocks for point reads): reads 16687 -> 18027/s (+8%), score +3.5% |
| 0023 | 22,389.4 | 29,479 | 17,005 | discard | fjall data block compression disabled: reads -6% (17005 vs 18027/s); larger blocks cost more than the LZ4 decode |
| 0024 | 23,365.8 | 29,417 | 18,559 | keep | per-segment cached StoreReader with a 4096-block cache for hit ids (tantivy default 100 blocks thrashed on big segments): reads +3%, score +2.0% (low-noise harness, kept) |
| 0025 | 26,424.9 | 28,941 | 24,127 | keep | single-clause booleans unwrapped so text searches run as TermQuery / term union and get tantivy block-max WAND pruning (filter applied in the pruning callback): reads… |
| 0027 | 27,031.5 | 28,298 | 25,822 | keep | cache maps behind RwLock (was a Mutex per segment per hit: 9% of read CPU in mutex waits natively) and one doc-store lookup per distinct segment per query: reads +7%,… |
| 0028 | 25,978.2 | 22,663 | 29,778 | discard | JSON docs stored in the tantivy doc store so find hits skip fjall: reads +15% but writes -20% (doc store + merges carry the JSON) and peak RSS 368 -> 507 MB; calibration… |
| 0033 | 27,000 | 26,541 | 26,890 | keep | tokio workers = available cores (1 in the container; was max(2)): score neutral (26715 / 27632 / 26654 vs 2-worker 27051 / 26790; the first env run's 30423 was an… |
| 0035 | 30,456.2 | 27,323 | 33,949 | keep | quick_cache LRU of hot documents (50k entries, invalidated on put/delete): reads 26890 -> 33949/s (+26%), score +12.8%; confirmed by a second run at 31372 (cal 647) |
| 0036 | — | 22,957 | 31,039 | crash | doc cache 150k entries: RSS gate failed (716 MB). Cause: cached Bytes come from Bytes::from_owner(fjall slice) and pin the whole 4 KB block each; 0035 had the same… |
| 0037 | 32,320.6 | 28,225 | 37,011 | keep | doc cache stores a compact copy instead of a Bytes pinning fjall's 4 KB block: peak RSS 309 -> 220-261 MB; runs 27496 (low outlier) and 32320 (best ever); mean 29.9k vs… |
| 0038 | 35,169.8 | 27,376 | 45,182 | keep | doc cache 150k compact entries (default): runs 35170 / 31069, mean 33.1k vs baseline mean 29.9k (+11%); RSS 310-337 MB |
| 0047 | 35,009.2 | 27,406 | 44,722 | keep | snapshot staging (index commit + hard links) moved after the segment upload, outside the WAL lock (snapshot may hold a little beyond its position; replay is idempotent):… |
| 0048 | 37,280.7 | 29,609 | 46,941 | keep | merge policy knob (HANUDB_MERGE default|lazy|none) + index_segments in status. default 37281 (best single run; reads 46.9k), lazy 27259 (reads 24.1k), none 27626 (reads… |
| 0050 | 30,589.8 | 30,392 | 30,789 | discard | commit interval 500 -> 2000 ms: runs 30590 / 31579 (mean 31.1k vs ~35k); writes best ever (30.4k, p99 4.9) but reads 31-33k (segments less merged when reads start) |
| 0052 | 33,913.5 | 28,043 | 41,013 | discard | eager LogMergePolicy (min_num_segments 4): runs 33914 / 34042 (mean 34.0k vs ~35k); very consistent reads (41.0k / 40.9k) but no gain |
| 0055 | 37,011.4 | 29,342 | 46,686 | discard | fjall block cache 64 -> 32 MB: runs 37011 / 30145 (mean 33.6k vs 37.0k; second run hit the low-read merge state); only -30 MB |
| 0058 | 33,954.8 | 29,399 | 39,217 | discard | idle merge kick (merge small segments 1 s after writes go quiet): runs 33955 / 34349 (mean 34.2k vs 37.0k), read p99 1.0 -> 3.8 ms; the merge competes with reads on the… |
| 0064 | 36,495.1 | 28,736 | 46,350 | keep | compact retries when a policy merge holds segments (the suite caught a flaky 400); scripts/smoke.sh passes 3/3. Off the hot path |
| 0066 | 36,367.6 | 28,138 | 47,004 | keep | METRIC: disk_read_mb / disk_write_mb in status (/proc/self/io) + an io log line every 10 s; bench.sh prints the last value before the crash. |
| 0071c | — | 2,073 | 4,868 | info | REAL e2-micro, 4 clients (what an app sees): write p50 1.7 ms / p99 4.8 ms, get p50 0.30 ms, find 0.43 ms, search 0.41 ms, read p99 1.2 ms |
| 0076 | 29,094.9 | 26,610 | 31,812 | keep | SAFETY: LogMergePolicy max_docs_before_merge 2M (segments beyond that are never merged: bounded merge memory on a 1 GB box). Bench unchanged (segments never reach 2M in… |
| 0087a | — | 2,703 | 7,858 | info | VM check after rolling 0087 out (e2-micro, twin mode, 10 s + 10 s, client on the VM): 2703 writes/s (p50 6.3 / p99 71 ms), 7858 reads/s (p50 2.6 / p99 13 ms), rss 193… |
| 0090 | 36,886.6 | 33,387 | 40,933 | discard | MEMORY/READS: fjall block cache 128 MB (was 64). Three runs: reads 45.9k / 42.7k / 34.2k (mean 40.9k vs the baseline's 40.1k), writes unchanged, peak rss +85 MB (519).… |
| 0093 | 37,717.3 | 33,962 | 42,041 | discard | DISK: LZ4 on every kv level (was: two youngest levels uncompressed). Disk writes 2.06 -> 1.76 MB per 1k docs (-15%), kv on disk 352 -> 151 MB, read-phase disk reads 250… |
| 0096 | 35,297 | 35,143 | 35,453 | keep | PROFILE-DRIVEN: perf on the write phase: tantivy indexing is the CPU (postings writer 9%, tokenizer 2.6%, fst 4%, TfAndPositionRecorder 2%), zstd 3%, kernel 18%; read… |
| 0101 | 37,426.7 | 35,049 | 40,130 | info | --merge eager re-test on the lsm_tree store: runs 35192 / 39661 - but 'eager' has no match arm (falls through to the default LogMergePolicy), so this was two more… |
| 0102 | 35,833.6 | 30,987 | 41,470 | keep | MERGES: a real --merge eager (LogMergePolicy min_num_segments 4, min_layer_size 10k, 2M cap): reads 40.8k / 42.1k (mid-band, no better than default's 45k runs) while… |
| 0103 | — | — | — | info | VM END-TO-END: scripts/remote-check.sh (API-only: bucket-durable batch insert, idempotency replay, get, search+filter with read-your-writes, $in + sort, stats aggs,… |
| 0105 | — | 22,659 | — | info | FOLLOWER UNDER LOAD (scripts/soak-follower.sh, 3 min mixed workload 60/30/10, leader on core 1, follower on core 2 tailing the same bucket with poll 200 ms): leader… |