HanuDB: A Cloud-Durable Document Database for Tiny Compute

Keywords: document database, object storage, write-ahead log, log-structured merge tree, inverted index, group commit, automated performance engineering, free-tier computing.

1Introduction

Small applications need a database that costs nothing while idle, survives the loss of its host, and answers interactively. Public clouds now offer both halves of a solution at no charge: a tiny always-free virtual machine and durable, regionally replicated object storage priced in cents per gigabyte-month. What is missing is a database that treats the object bucket, not the machine's disk, as the source of truth while still serving reads and writes at local-disk latency.

Two existing families approach this from opposite sides. Bucket-native storage engines such as SlateDB [1] place the log-structured merge tree itself in object storage; they are durable by construction but pay object-store latency on every cache miss and tens of seconds to open. Replication tools such as Litestream [2] ship a local SQLite database's write-ahead log to a bucket; they keep local latency but inherit SQLite's write model and offer no search. Prior measurements by the author on the same virtual machine class placed a local engine with a shipped log at ten-microsecond reads and one-second takeovers, against hundred-millisecond misses and ten-second opens for the bucket-native approach [3]. That comparison fixed HanuDB's architecture: serve from local disk, make the bucket the durable home, and never let a read touch the bucket.

The second theme of this paper is how the engine was built. Rather than a hand-tuned design, HanuDB was produced by an autonomous experiment loop. A fixed benchmark and correctness suite define the objective, a set of hard gates defines what is inadmissible, and an agent proposes one change at a time, measures it in a container shaped like the target machine, and keeps or discards it on the evidence. Over 108 numbered experiments, 74 were kept, 27 discarded and 3 crashed. The loop found the changes one would expect, such as filter caches and group-commit tuning, and several one would not, including that a second thread on a single shared core is a net loss, that the allocator determines how much memory a rebuild keeps, and that resident set size is the wrong quantity for an out-of-memory guard inside a control group.

This paper makes the following contributions:

• A design for a document database with local-latency reads whose durable state is entirely in an object bucket, including a single-log durability model with per-engine watermarks (§3.3), a hash-chained segment format with a create-if-absent fencing rule (§3.4), and physical snapshots that make recovery proportional to the log tail (§3.5).
• An automated optimization methodology with a fixed workload, hard admissibility gates and a noise-aware acceptance rule (§4), and an account of what it found.
• Measurements on a one-core harness, in multi-hour soaks up to 14 million documents, and on a real free-tier machine with real buckets in two regions (§5).
• A catalogue of negative results and design lessons for engines that must share one core with their own background work (§6).

2Related Work

Object storage as the durable home. Litestream [2] and Turso [4] stream SQLite's write-ahead log to object storage and restore from it; HanuDB adopts the same posture with a document model, full-text search and a hash chain over the shipped segments. SlateDB [1] and similar bucket-native LSM trees make object storage the primary store and are the natural choice at terabyte scale; on a one-gigabyte machine whose working set fits on disk, local serving wins by two orders of magnitude on read latency. Neon [5] separates compute from a distributed storage tier that plays the same role as HanuDB's bucket at far greater scale and cost; HanuDB's point-in-time restore is a small approximation of Neon's branching.

Storage engines. The document store is lsm_tree, the log-structured merge core of the fjall key-value engine [6], used without fjall's own journal. The index is tantivy [7], a Lucene-derived search library whose JSON fields give a schemaless index with per-path fast columns. Both are pure Rust, compile quickly and have small memory footprints, which mattered as much as their throughput.

Write-ahead logging and group commit. The group-commit fsync loop follows the standard design [8]; the contribution here is in the file handling: a preallocated, zero-filled log written positionally so that fdatasync never flushes metadata, reused without truncation, with a decoder that tolerates stale, torn and zeroed tails.

Automated performance engineering. Search-based and autotuning approaches to systems performance share one shape: a fixed metric, a read-only benchmark and a rule that decides whether a candidate survives. Applying that shape to a storage engine adds a constraint those approaches rarely carry, because a faster engine that loses a write is not a faster engine: correctness gates (recovery, byte-identical documents, working search) and resource gates (memory, bucket operations) sit alongside the score, and a candidate that fails one is inadmissible whatever it scored.

3System Design

3.1Overview

HanuDB is one process (Figure 1). An HTTP server accepts JSON documents into named collections. Each write is appended to a write-ahead log, applied to the document store and handed to an indexer thread. A sealer thread periodically compresses the pending log into a segment object and uploads it to one or two buckets; a snapshot task periodically packs the on-disk engine state into a snapshot object. A fresh machine restores itself from the newest snapshot and the segments after it. Reads are served entirely from local state.

3.2The write path and durability levels

A request body is parsed once. Under a short lock the engine assigns a monotonically increasing transaction number, encodes the record (transaction, kind, collection, identifier, document bytes) and writes it at the next offset of the open log file, then applies the document to the store's memtable. The record is passed to a dedicated indexer thread through a bounded channel. The handler then waits on a watch channel for the durability level the client requested:

• none: acknowledged after the in-memory apply; the log is written but not synced.
• local (default): acknowledged after the next group commit. One fsync loop wakes on demand and issues a single fdatasync for every writer waiting at that moment. This level survives a process crash.
• bucket: acknowledged when the segment containing the record is in the bucket, or in both buckets in twin mode. This level survives the loss of the machine.

The log file is preallocated and zero-filled once, and records are written positionally rather than appended, so the file's size and extent map never change and fdatasync flushes data blocks only. After each seal the same file is reused from offset zero without truncation. Recovery tolerates both properties: the decoder stops at the first zeroed header, torn record or transaction discontinuity, and discards records whose transaction number lies below the next transaction of the already-sealed segments. On the harness these two changes together yielded 10% more writes per second and 15% fewer bytes written to disk (§5.1). An alternative using fallocate was measured and rejected: ext4 converts each unwritten extent on first write, which returns metadata to every fdatasync and cost 19% of write throughput.

3.3One log, two watermarks

The single most consequential design change was the removal of a second write-ahead log. The fjall engine journals every insert before its memtable, so with HanuDB's own log in front of it each document reached the disk three times before its first compaction. HanuDB now uses lsm_tree directly and the document store keeps no journal: its memtable is lost in a crash and rebuilt from HanuDB's log.

Correctness rests on two durability watermarks. The store records kv_flushed_txn, the newest transaction whose documents reside in a flushed table. The index records index_txn, the newest committed transaction, stored inside tantivy's own commit payload so that it is atomic with the commit. On boot the engine replays the log past each engine's own watermark, so each side receives exactly the records it lacks: at most one memtable's worth for the store and one commit interval's worth for the index. The floor below which open-log records are treated as stale follows the smaller of the two. Snapshots and clean shutdown flush the memtable first. The change reduced disk writes by 20%, resident memory by 80 MB and produced the best benchmark score of the study (§5.1).

A subtle failure mode was found by the correctness suite shortly after this change and is worth recording. For a few builds the staleness floor followed the index watermark alone, which after a forced kill discarded records the store had not yet flushed. A test that kills the process and verifies every acknowledged document caught it once in three runs; the test now forces the vulnerable window open on every run by waiting for an index commit before the kill. The general rule is that every "how far have we got" quantity in a system with several durability watermarks must follow the minimum.

3.4Segments and the hash chain

Every second, or every 4 MB, the sealer compresses the pending records with zstd and writes one segment object named by its sequence number. The segment header carries a magic and version, the sequence, the first transaction, the record count, a timestamp and the blake3 hash of the previous segment; the object ends with the blake3 hash of everything before it. A reader verifies the chain from the snapshot forward and rejects any gap, reorder or corruption.

Segment objects are created with the bucket's create-if-absent precondition. A second writer for the same prefix therefore collides on its first upload and fences itself by exiting with a distinct status. This rule, and not any lease or clock, is the system's safety property against split brain; the lease described in §3.7 provides only liveness.

Cost follows directly from the cadence. One segment is one class-A bucket operation. With a one-second cadence the worst case is one operation per second of write activity and idle costs nothing; reads never touch the bucket. The benchmark measures 0.13 class-A operations per thousand writes.

3.5Physical snapshots and recovery

A logical snapshot, meaning every live document re-serialized, would make recovery time proportional to data size because every document would have to be re-indexed. HanuDB instead takes physical snapshots of the engine directories. Every 200 segments, and only when the log since the last snapshot is at least half the size of the on-disk state, the engine commits the index, flushes the memtable and, under the write lock, hard-links the immutable index and store files into a staging directory; this takes milliseconds. Outside the lock, on its own task and on a thread niced to 10, it tars and zstd-compresses the staging directory into a file, syncing every 32 MB, and uploads it as a multipart object. Each tar entry is cut at the size recorded in its header, because a hard-linked table that the flush thread is still writing can grow between listing and copying.

A rebuild downloads the newest snapshot to a file, extracts it entry by entry with the same periodic sync, opens both engines on the result and replays only the segments after the snapshot's sequence. Retention deletes snapshots older than the third-newest and every segment at or below the oldest retained snapshot, so the bucket holds about three snapshots and at most a few hundred segments. Point-in-time restore selects the newest snapshot at or before a requested instant and replays segments up to it into a fresh directory without writing to the bucket.

3.6Index and query execution

The index schema is dynamic in the sense of Elasticsearch's dynamic mapping. Every field whose value is at most 64 characters is indexed as an exact keyword and a fast column, which serve filters, sort and aggregations; longer strings are not indexed as keywords, following Elasticsearch's ignore_above rule. All string values in a document are concatenated into one text field per collection, which serves free-text search. Field paths are namespaced by collection so that no per-query collection term is needed. A Mongo-style filter language with nested paths, ranges, set membership, boolean combinators and negation compiles to tantivy queries.

Query execution relies on four caches, each keyed by segment identifier and safe because tantivy segments are immutable. (i) Equality filters become per-segment bitsets built once from postings and applied inside a query wrapper, so that tantivy's block-max WAND pruning continues to operate; this is the filter cache of Elasticsearch. (ii) Sort columns are opened once per segment rather than per query. (iii) Document-store readers are cached per segment with a larger block cache than tantivy's default. (iv) A 150 000-entry LRU holds compact copies of hot documents; copies rather than zero-copy slices, because a slice pins the whole underlying block. A further rule matters for text queries: block-max WAND engages only for a bare term query or a pure union of terms, so single-clause boolean wrappers are peeled before execution.

3.7Twin buckets, followers and leases

In twin mode every segment and snapshot is written to two buckets in different regions in parallel, and a write is bucket-durable only when both hold it. A rebuild lists both buckets, reads from the first with fallback to the second, and re-mirrors one-sided objects in both directions. Cost is exactly twice the class-A operations.

A follower boots from the bucket like any node and then polls for the next segment every second, verifying the chain and applying it. It serves every read endpoint and answers writes with HTTP 421 so that a client falls back to the leader. With the lease option every node boots as a follower; the one that creates the lease object becomes the writer and renews it by compare-and-swap at a third of the time-to-live. A standby that observes an expired lease takes it by compare-and-swap and promotes itself in place. A leader that fails to renew fences itself and exits. Measured locally with a three-second time-to-live, takeover completed three seconds after a kill signal, twice in succession, with the chain verified.

The API is completed by bearer-token authentication, CORS, body limits, idempotency keys stored as replicated documents inside the same transaction, read-your-writes through a transaction number that the client passes back to a search, Prometheus metrics, a chain-verifying audit command, an offline dump, and a live export of zstd-compressed JSON lines that analytical engines such as DuckDB read directly from the bucket.

4Methodology: Automated Optimization

The engine was developed by an autonomous experiment loop. A written program specifies the objective, the editable and read-only parts of the tree, the admissibility gates and the acceptance rule; an automated agent executes the loop without human intervention, one change per iteration.

4.1Workload and score

The benchmark client is fixed for the whole study. Thirty-two concurrent HTTP clients each insert one document of roughly 450 bytes per request for 20 s at the local durability level. Documents carry a user identifier drawn from 1 000 values, a kind, two tags from 50, a timestamp and a title and body drawn from a 3 000-word vocabulary with a skewed distribution. Thirty-two clients then read for 20 s: 50% get-by-identifier, 25% filtered find sorted by timestamp with a limit of 20, and 25% full-text search with a filter and a limit of 10. Finally the server wipes its local directory and rebuilds from the bucket; every write it had acknowledged as bucket-durable must return byte-identical on a sample, and search must work. The score is the geometric mean of writes per second and reads per second, so that neither side can be traded for the other.

4.2Harness

The server runs in a Linux container pinned to one physical core with a 768 MB memory limit, as a stand-in for the target machine, on an Apple M4 Max host. The bucket is a local directory with simulated latency of 70 ms per PUT and 40 ms per GET, the same-region floor measured from a Google Compute Engine instance. Two harness decisions were themselves the result of experiments. A CFS CPU quota throttles in 100 ms periods and produced 50 ms tail latencies and 8% run-to-run noise, so the container is pinned to a core instead. Running the client on the host added 0.6 ms per request through the container port proxy and capped writes at 18 000/s, so the client runs in a second container on the server's loopback. Each run additionally times a fixed hash computation on the server core; runs on an overloaded host are discarded.

4.3Gates and acceptance rule

A change is inadmissible, with a score of zero, if any gate fails: the rebuild must succeed and verify; peak resident memory must not exceed 600 MB; class-A bucket operations must not exceed five per thousand writes; there must be no request errors; find and search must return hits. Tail latency is reported rather than gated, but a change that doubles a 99th-percentile latency for a marginal gain is rejected.

Identical builds scored 27 500 and 32 300 in consecutive runs, because the read phase runs over whatever the write phase produced and in whatever merge state the index is in when reads begin. Every candidate is therefore run twice and its mean compared with the baseline's mean; a keep requires a gain above 3%, or equal speed with less code or less memory. Simplicity is an explicit criterion of the program: a 1% gain for a hundred lines is a discard, and an equal score with fewer lines is a keep. Each accepted change is stored as a numbered patch together with the benchmark log that justified it, so that the full lineage of the engine is reproducible.

4.4Correctness and long-running checks

Before every keep a native correctness suite must pass. It grew from 41 to 59 checks over the study and covers authentication, create-read-update-delete, filters, text search, sort, aggregations, idempotency, read-your-writes, restart after a kill signal, rebuild from the bucket, loss of one twin bucket, snapshots, compaction, chain verification, resynchronization, dump and TLS. Roughly every twenty experiments the loop also runs two checks that a 20 s benchmark cannot express: a scale run that writes about 2.5 million documents and rebuilds, and a ten-minute soak that records memory every ten seconds and then performs the crash drill. Four of the soaks in this study failed, each for a different reason (§6), and each failure became a design change.

5Evaluation

5.1Harness results

Figure 2 plots the score of every kept experiment. Table 1 lists the milestones. The curve has three regimes. Experiments 1 through 6 fixed the harness and the memory model; experiments 8 through 53 removed per-query work from the read path, each step being the elimination of one cost; experiments 57 through 108 added product features and safety mechanisms whose requirement was to keep the score, not to raise it, with the exception of the allocator change and the removal of the second journal.

Writes and reads improved for different reasons. Writes are bound by the group-commit fdatasync, measured at 0.55 ms on this disk, and by HTTP framing on the single core; they rose in three steps, from the harness correction, the log file handling and the allocator, and were otherwise flat. Reads rose in many steps because each was the removal of one unit of per-query work: the collection term, the posting-list intersection for filters, the per-query column open, a mutex per segment per hit, document-store block thrashing and finally the store lookup itself for hot documents. Profiling of the final build attributes 18% of write-phase samples to the kernel, roughly 20% to tantivy indexing and 3% to compression; the read phase is dominated by socket wake-ups, per-segment term lookups, block-max WAND evaluation and integer decoding.

5.2Scale and soak

The ten-minute soak wrote 14 million documents with zero errors while seven snapshots of up to 4.5 GB were packed in the background; anonymous memory stayed within a 100 MB band for the whole run. In the subsequent crash drill the 4.5 GB snapshot was extracted in 10.6 s and a 474 000-record tail replayed, 39.7 s in all, and every document returned byte-identical. Reads over 14 million documents, far beyond the page cache, fell to 1 478/s, which is the disk's random-read rate rather than the engine's. The update-heavy soak, with 40% updates and 10% deletes, exercised tantivy deletions, store tombstones and compaction; the live count matched exactly and memory stayed flat.

5.3A real free-tier machine

The same binary was deployed to an e2-micro in us-central1 with a real bucket and, in twin mode, a second bucket in us-east1. The benchmark client shared the machine's two shared vCPUs, so these figures include client cost.

The last row is the honest limit of the free tier and, in our view, the most useful measurement in this paper. An e2-micro bursts to two vCPUs but sustains a quarter of one; once the burst credit is spent, the engine's background work (merges, compaction, snapshot packing, uploads) competes with request handling for that quarter core. The second limit is the disk. The engine writes about 3.7 MB to disk per thousand documents across the log, the store, the index and the local segment copies, so a few thousand writes per second exceed what a 30 GB standard persistent disk sustains, and once the working set outgrows the page cache, reads queue on the same disk: after a two-minute cooldown writes recovered to 4 386/s but reads stayed at 1 575/s with 65% of CPU time in I/O wait. Bursty traffic is served well; sustained ingestion or a hot set above roughly 400 MB requires the next machine size and a balanced disk, together about fifteen dollars per month. A compression option that applies LZ4 to every store level, reducing bytes written by a further 15%, is enabled on the deployed machine for this reason.

5.4Cost

After the first day of operation, during which every measurement above was run against the deployed machine, each bucket held 211 MB and about 150 class-A operations had been billed: roughly one cent, with the machine and its disk free. Under a nominal load of ten thousand writes per day arriving in bursts, segment cadence bounds the bill below ten cents per month.

6Discussion: What the Loop Found

The negative results shaped the engine as much as the accepted changes. We list those that reversed an assumption or changed the design.

• On one core every background thread competes with request handling. Moving segment sealing off the write lock onto a spare thread reduced writes by 13%. Blocking writers for microseconds is free; additional CPU is not. Eager index merging and an idle-time merge trigger both lost for the same reason, even though merges themselves are worth their cost: without them reads halve.
• Two write-ahead logs are one too many. Removing the store's journal in favour of per-engine watermarks (§3.3) was the largest single improvement in throughput, disk writes and memory.
• Bytes written matter more than CPU on a slow disk. The zero-filled positional log, the reuse of the log file, a leveled compaction with a larger level-0 threshold, larger memtables and the store compression option together reduced disk writes per thousand documents from 4.5 MB to 1.8 MB. Skipping JSON re-serialization and the indexer's re-parse freed CPU but changed nothing, because writes are bound by fdatasync.
• Resident set size is the wrong quantity for an out-of-memory guard. It counts clean memory-mapped index pages that the kernel reclaims under pressure. What kills a process inside a control group is anonymous memory plus dirty page cache, and dirty pages count against the limit, so every path that writes gigabytes must sync periodically. Three separate such paths were found, each by a soak being killed: snapshot packing, snapshot extraction and the upload to a local bucket.
• Never run a minute-long job inline in a control loop. Snapshot packing originally ran in the sealer loop, so for up to 130 s per snapshot nothing was uploaded, the memory guard did not execute and clients timed out. Moving it to its own task turned 213 000 client timeouts into zero, and on one core the pack must also run at reduced priority.
• Read the storage engine's worker model before trusting its limits. The fjall engine sized its worker pool at one thread on the target machine and ran compactions on it, so a long compaction blocked every memtable rotation and the active memtable grew without bound at replay speed. This was the cause of every rebuild out-of-memory failure in the soaks. Two workers and a replay throttle bounded the write buffer at 75 MB.
• The allocator decides how much of a rebuild you keep paying for. With glibc malloc, an in-process rebuild of 4.9 million documents settled at 585 MB of anonymous memory where a fresh process needs 229 MB, because freed pages remained in per-thread arenas. jemalloc with a one-second decay settled at 245 MB and improved write throughput by 22%. An earlier trial of mimalloc had been rejected only because its rebuild peak was compared with glibc's understated live figure; the measurement, not the allocator, was wrong.
• Block-max WAND has preconditions. tantivy prunes top-k evaluation only for a bare term query or a pure union of terms. Wrapping the text clause in a boolean query with one MUST clause silently scores every match; for common terms that was the whole search cost. Peeling single-clause booleans gave 30% more reads.
• Filter caches are worth building twice. Evaluating an equality filter on a fast column instead of intersecting postings was worth 34%; a cached per-segment bitset applied inside the query, so that pruning stays active, was worth a further 61%.
• Copy what you cache. A zero-copy slice over a store block, once placed in the document cache, kept the entire 4 KB block resident; 150 000 such entries exceeded the memory gate. Compact copies halved the cache's footprint.
• Fewer, larger index segments are worth their merge CPU, but the timing is uncontrollable. The run-to-run read variance of the benchmark, from 27 000 to 47 000 reads/s, is entirely the merge state of the index when the read phase begins. The acceptance rule of §4.3 exists because of this.

7Limitations

HanuDB is a single-writer system; its write ceiling is one core and one disk's fsync rate. It offers atomic multi-document batches but no interactive transactions across requests, no secondary indexes beyond the dynamic keyword and fast columns, no query planner and no sharding. A hot set larger than memory degrades to the disk's random-read rate. Field-scoped full-text search is not indexed, only the joined text. Automatic failover requires a second machine, which is not free. The harness core is faster than the target machine's, so harness figures are an upper bound; the free-tier measurements in §5.3 are the numbers an application should plan against. Finally, the optimization loop measured a single insert-heavy workload; the update-heavy soak exists precisely because the scored benchmark does not exercise deletions.

8Conclusion

A document database can keep local-disk latency while making an object bucket its only durable home, on the smallest machine a cloud offers for free, if three things hold: one write-ahead log serves every engine, with per-engine watermarks in place of per-engine journals; the log is shipped as hash-chained segments whose creation is itself the fencing mechanism; and recovery is proportional to the log tail through physical snapshots. An autonomous experiment loop with a fixed workload and hard gates was sufficient to take such an engine from a first draft to 34 800 writes/s and 46 200 reads/s on one core, and its failures under long-running load taught more than its successes: on a shared core, the engine's own background work is its principal adversary, and memory must be accounted the way the kernel accounts it. The implementation, the benchmark, the correctness suite, the experiment log with every kept and discarded change, and the deployment scripts accompany this paper, released under the MIT licence at github.com/madhudream/hanudb.

References

1. SlateDB contributors. SlateDB: an embedded key-value store built on object storage. https://slatedb.io, 2024–2026.
2. B. Johnson. Litestream: streaming replication for SQLite. https://litestream.io, 2021–2026.
3. Madhu. twinlog: a two-bucket write-ahead log protocol; measurements of GCS latency from a GCE instance and a comparison with bucket-native engines. Technical notes, September 2026.
4. Turso contributors. Turso database and libSQL. https://turso.tech, 2023–2026.
5. Neon. Neon: serverless Postgres with separated storage. https://neon.tech, 2022–2026.
6. M. Fenner et al. fjall and lsm_tree: an LSM-based embedded key-value storage engine in Rust. https://github.com/fjall-rs/fjall, 2023–2026.
7. P. Masurel et al. tantivy: a full-text search engine library in Rust. https://github.com/quickwit-oss/tantivy, 2016–2026.
8. D. J. DeWitt, R. H. Katz, F. Olken, L. D. Shapiro, M. R. Stonebraker and D. A. Wood. Implementation techniques for main memory database systems. In Proc. ACM SIGMOD, pp. 1–8, 1984.
9. S. Ding and T. Suel. Faster top-k document retrieval using block-max indexes. In Proc. ACM SIGIR, pp. 993–1002, 2011.
10. P. O'Neil, E. Cheng, D. Gawlick and E. O'Neil. The log-structured merge-tree (LSM-tree). Acta Informatica 33(4):351–385, 1996.