Writes: the log, group commit and the disk

What one fsync, one core and a few dozen IOPS allow

The question

Of 37 attempts in this theme, which ones survived two benchmark runs, the correctness suite and the gates?

adopted

With two durability watermarks, every "how far are we" number must follow the smaller one. After 0091 the current.log staleness floor (meta.next_txn) followed the index watermark, so after a SIGKILL the records the kv had not flushed yet were dropped as…

Attempts37

Kept13

Discarded or crashed13

Best kept score40,116.4

Every run in this theme

Every run in this theme
#	score	writes/s	reads/s	verdict	what changed
0001	9,046.8	8,747	9,357	keep	baseline: fjall + tantivy(t,k,all) + WAL group commit, flush 1s, commit 500ms
0003	8,784.2	8,347	9,244	discard	seal segment outside the wal lock (spawn_blocking zstd+fsync): writes -13% both runs (8784 / 8972); on 1 CPU the extra thread only competes
0006	12,273.3	25,421	5,926	keep	HARNESS: bench client now runs inside the VM on the server loopback (cores 2-5); reference run of the 0005 code. Not comparable with earlier scores.
0012	—	13,355	8,853	crash	INVALID (ran on the 0001 code by mistake: keep.sh had silently never updated .baseline, discard.sh reset the tree): doc store Compressor::None + 4 KB blocks; re-test…
0020	20,581.8	29,162	14,526	discard	doc store Compressor::None + 4 KB blocks (valid re-test): +1.1% (noise); LZ4 decode of the id block is not a bottleneck in the container
0026	21,998.2	16,276	29,732	discard	fsync group-commit window 200 us: writes -44% (tokio sleep granularity is ~1 ms, so the window adds a full ms of latency; writes are latency-bound with 32 clients).…
0029	26,895.2	24,423	29,617	keep	fjall docs memtable 64 -> 16 MB: peak RSS 368 -> 221 MB; +3.6% vs the baseline re-run under the same host load (calibration 664 vs 687; quiet-host baseline was 27031)
0030	26,773.4	23,976	29,898	keep	get returns fjall's slice zero-copy (Bytes::from_owner): score unchanged (-0.5%, noise, calibration 675); kept as a simplification
0033b	6,170.9	4,188	9,093	info	REAL e2-micro hanudb-1 (client on the VM, real GCS): writes fsync-bound on pd-standard (p50 6.9 ms), 41922 docs rebuilt from GCS in 2.6 s, 0 mismatches
0034	26,807.4	27,706	25,938	discard	TCP_NODELAY on accepted sockets (ListenerExt::tap_io): no change (loopback client); not simpler
0040	33,163.4	29,620	37,131	discard	hand-rolled hyper http1 loop (pipeline_flush, writev, nodelay): runs 33163 / 32515, mean 32.8k vs baseline mean 33.1k: no gain, more code
0042	31,886.7	26,419	38,486	discard	batched WAL writes (records buffered, one write syscall per group commit): SIGKILL durability OK but runs 31887 / 36001 (mean 33.9k) vs baseline 34938 / 35619 (mean…
0046	36,931.3	29,263	46,609	keep	WAL file preallocated + zero-filled (positional writes never change the file size, fdatasync flushes data only; lenient log decode stops at zeros/torn/txn gap): writes…
0046b	—	3,699	10,677	info	REAL e2-micro twin mode with the preallocated WAL: write p50 8.1 -> 5.4 ms (3.7k/s); write p99 85 ms = snapshot staging under the WAL lock (next experiment)
0047b	—	5,543	7,472	info	REAL e2-micro with 0047: write p99 85 -> 21.6 ms, 5.5k writes/s (p50 4.6 ms)
0051	32,718	29,951	35,740	discard	commit interval 500 -> 250 ms: runs 32718 / 30385 (mean 31.6k vs ~35k). 500 ms stays
0059b	—	—	—	info	REAL e2-micro: aggregation (terms + stats) over 41922 docs in 64 ms on the VM
0062	36,969.8	29,097	46,973	discard	skip JSON re-serialisation when the document carries _id: runs 36970 / 37107 (mean 37.0k = baseline); writes are fsync-bound, freed CPU changes nothing; not simpler
0061b	—	—	—	info	REAL e2-micro rolled to 0061 (idempotency keys + TTL, aggregations, dump, compact); status OK
0065c	—	4,386	1,575	info	REAL e2-micro after cooldown: writes back to 4.4k/s (burst credit recovered, steal negligible) but reads 1.6k/s p99 232 ms with 65% iowait: 30 GB pd-standard = a few…
0067	33,010.4	22,839	47,711	discard	fallocate instead of zero-fill for the WAL: disk writes 4.54 -> 3.36 MB per 1k docs (-26%) but writes 28k -> 22.8k/s (p50 0.69 -> 0.95 ms): ext4 converts unwritten…
0068	36,484.5	28,379	46,904	keep	WAL file reused from offset 0 after each seal (no truncate, no re-zeroing; recovery ignores records with txn below the sealed segments' next txn): disk writes 4.54 ->…
0068b	37,001.6	29,169	46,938	info	disk attribution after a 584k-doc run: written 2229 MB; on disk after rebuild index=105 MB kv=315 MB seg=1 current.log=5 bucket=77 MB. fjall (journal+flush+compaction)…
0069	36,401.9	28,482	46,524	keep	fjall Leveled compaction with l0_threshold 8 and 128 MB table target: disk writes 3.84 -> 3.35 MB per 1k docs (-12.5%), score neutral (33265 / 36402), get p50 unchanged.…
0070	36,621	28,767	46,619	discard	fjall journal LZ4 compression: disk writes 3.34/3.32 vs 3.35 MB per 1k docs = no change; score neutral
0070b	—	—	—	info	disk-write attribution: no merges 3.11 MB/1k (merges ~0.24), commit 60 s 3.29 (tantivy flushes happen anyway); tantivy total ~0.4, WAL+local segment+bucket copy ~0.7,…
0071	35,789.3	28,920	44,291	keep	fjall memtable 16 -> 64 MB (knob --kv-memtable-mb, default 64): disk writes 3.35 -> 2.65 MB per 1k docs (-21%, fewer flushes/compactions), score neutral (34526 / 35789),…
0071b	—	4,450	2,410	info	REAL e2-micro (0071 build): 30 s at 4450 writes/s wrote 489 MB to disk = 3.66 MB per 1k docs = ~16 MB/s, above what a 30 GB pd-standard sustains -> bursts only, as…
0072	—	—	—	keep	HARNESS: bench client keeps every txn (8 B) and a 1-in-64 document sample instead of every document; ids regenerated as w{client}-{k}. Same requests, same gates; long…
0074	33,135.8	28,314	38,779	keep	SAFETY: snapshot packing fdatasyncs every 32 MB so a multi-GB pack never holds more dirty pages than that in the cgroup. Bench unchanged; soak with growth 0.25 re-run…
0078b	30,182.3	27,024	33,710	info	valid regression run of 0078 after killing the stray soak containers (calibration 564)
0087c	—	—	—	info	ROOT CAUSE of every rebuild OOM: segments-only rebuild of 3.8M docs logged wb_mb=88 -> 317 -> 542 with sealed=1 (rebuild progress line every 50 segments). fjall rotates…
0088	36,585.2	33,524	40,077	keep	SAFETY+SPEED: fjall worker_threads(2) (default was min(cores,4) = 1: a compaction blocked every memtable rotation and flush behind it, so the active memtable grew…
0089	35,240.7	33,729	36,827	discard	CPU: hand the HTTP handler's parsed serde_json::Value to the indexer (Arc in the Op / IndexMsg) so the indexer does not re-parse the document. Writes 33.1k / 34.3k vs…
0091	40,116.4	34,821	46,218	keep	ARCH: the document store uses lsm_tree directly (src/kv.rs) instead of fjall: no second write-ahead journal (HanuDB's WAL already is one), our own flush + compaction…
0099	35,065.7	35,086	35,045	discard	WRITES: a precise group-commit window (spin on the blocking thread, 150 / 300 us) before each fdatasync: writes 35.2k / 35.0k (unchanged), p99 6.0 ms (was 4.7).…
0107	35,469.4	35,274	35,674	keep	DISK: --kv-compress warm ([None, Lz4]: only the youngest level uncompressed). On the bench it equals cold (kv 299-357 MB, disk writes 1436-1469 MB): with l0_threshold 8…

What we found

With two durability watermarks, every "how far are we" number must follow the smaller one. After 0091 the current.log staleness floor (meta.next_txn) followed the index watermark, so after a SIGKILL the records the kv had not flushed yet were dropped as stale: a silent data-loss window that a smoke check caught only once in three runs. The check now waits for an index commit before the kill so the window is always open (0092).
Two write-ahead logs are one too many. fjall journals every insert before the memtable; HanuDB already has a WAL, so each document was written three times before its first compaction. Using lsm_tree directly (0091) with our own flush/compaction threads and a per-side durability watermark (kv: newest txn in a flushed table; index: the txn stored in tantivy's commit payload) cut disk writes 20 %, resident memory 80 MB, and gave the best score so far. Recovery replays the WAL past each side's own watermark, so the memtable that is lost in a crash costs one memtable of replay.
Read the storage engine's worker model before trusting its limits. fjall 3 rotates a full memtable through a message to a worker pool sized min(cores, 4) — one thread on the target box — and the same thread runs compactions, so a long compaction blocks every rotation behind it and the active memtable grows for its whole duration; max_write_buffer_size is stored and never enforced. At replay speed that was ~600 bytes of memtable per replayed document (542 MB after 1.2 M records) and the cause of every rebuild OOM in the soaks. Two workers plus a replay throttle keep the write buffer under 75 MB (0088). Calling rotate_memtable() yourself deadlocks (it holds the journal lock while sending to the worker queue).
On a disk with a few dozen IOPS, bytes written matter more than CPU. fallocate looks free but ext4 converts each unwritten extent on first write, so fdatasync is no longer data-only. A zero-filled file written once and reused forever is the cheap option; the recovery decoder has to ignore stale records (txn below the sealed segments' next txn) for that to be safe.
Appending to a WAL makes every fdatasync also flush the inode (size change). A preallocated, zero-filled file written at an offset turns fdatasync into a data-only flush: +10 % writes even on a virtual disk; the recovery decoder must then stop at zeros and torn tails.

hanudbwritesautonomous loop