A database that lives in a bucket
Kansas City · Independent · since 13 September 2026 · updated 16 September 2026
Abstract
HanuDB serves from local disk and keeps nothing that matters there. Delete the machine and a new one rebuilds from the bucket alone. An agent tuned it over 108 experiments while I was asleep.
Keywords: databases, rust, object storage, automated optimisation.
Small applications want a database that costs nothing when idle, survives losing its host, and still answers quickly. Clouds already give away both halves: a tiny always-free virtual machine, and durable replicated object storage priced in cents per gigabyte-month. What was missing was something that treats the bucket, not the machine's disk, as the truth, while still serving at local-disk latency.
The two existing approaches sit at opposite ends. Bucket-native engines put the storage tree itself in object storage: durable by construction, and paying object-store latency on every cache miss and tens of seconds to open. Log-shipping tools keep a local database and stream its write-ahead log to a bucket: local latency, inherited write model, no search. My own earlier measurements on this machine class put a local engine with a shipped log at ten-microsecond reads and one-second takeovers, against hundred-millisecond misses and ten-second opens the other way. That settled the architecture before any code was written.
The engine is called HanuDB, after Hanuman, who was small and carried a mountain. It runs on the smallest thing a cloud gives away.
| measure | value | conditions | source |
|---|---|---|---|
| Benchmark score | 9 047 → 40 116 | √(writes/s × reads/s), on a 1-core, 768 MB harness | results.tsv |
| Writes / reads per second | 34 821 / 46 218 | — | run.log of experiment 0091 |
| Experiments | 108 | 74 kept, 27 discarded, 3 crashed | results.tsv |
| Soak | 14 M documents | ten minutes, zero errors | soak-server.log |
| Rebuild from the bucket | 40 s | 4.5 GB snapshot plus tail | README.md, Scale |
| Cost to run | ≈ $0.01 / day | measured on a real free-tier machine | README.md, Real e2-micro |
1The design, in the parts that were not obvious
One process. An HTTP server takes JSON documents into named collections; each write is appended to a write-ahead log, applied to the document store and handed to an indexer. A sealer compresses the pending log into a segment and uploads it to one or two buckets. A snapshot task packs the on-disk state periodically. A fresh machine restores from the newest snapshot plus the segments after it. No read ever touches the bucket.
Three decisions carried the design. One write-ahead log serves both engines, with a durability watermark per engine instead of a journal per engine, so recovery replays exactly the records each engine is missing. The shipped segments are hash-chained and their creation is itself the fencing mechanism — a create-if-absent on the next sequence number is how two machines discover they both think they are the writer. And physical snapshots make a rebuild proportional to the log tail rather than to the size of the data.
Removing the document store's own journal in favour of those watermarks was the single largest improvement in the project, in throughput, in bytes written and in memory, all at once. Two write-ahead logs were one too many and it took me a while to see it.
2How it was tuned
I did not hand-tune this. A fixed benchmark ran the server inside a container shaped like the target machine — one core, 768 MB — and scored the geometric mean of writes and reads per second under 32 clients. Anything that broke recovery from the bucket, exceeded the memory limit, corrupted a document or cost too many bucket operations scored zero regardless of speed. An agent proposed one change at a time and kept or discarded it on the result.
The acceptance rule exists because of a measurement problem. Two identical builds differ by about eight per cent, almost entirely because of what state the index merger happens to be in when the read phase starts — the read figure alone swings between 27,000 and 47,000 per second. So every candidate got two runs, and was kept only if the mean of both beat the baseline by more than three per cent, or made the code simpler at equal speed.
108 numbered experiments: 74 kept, 27 discarded, 3 crashed. The score went from 9,047 to 40,116, which is 34,821 writes and 46,218 reads per second on that one-core harness.
3The findings that reversed what I expected
Several of these I would have bet against:
- On one core, every background thread is competing with request handling. Moving segment sealing off the write lock and onto a spare thread cost 13% of writes. Blocking a writer for microseconds is free; another thread is not.
- Bytes written matter more than CPU on a slow disk. A zero-filled positional log, reusing the log file, larger memtables and store compression together cut disk writes per thousand documents from 4.5 MB to 1.8 MB. Separately, skipping JSON re-serialisation freed real CPU and changed the score by nothing, because writes are bound by fdatasync.
- Resident set size is the wrong quantity for an out-of-memory guard. It counts clean memory-mapped index pages the kernel will happily reclaim. What kills a process inside a control group is anonymous memory plus dirty page cache — and dirty pages count, so every path writing gigabytes has to sync as it goes. Three such paths were found, each by a soak being killed.
- The allocator decides how much of a rebuild you keep paying for. Under glibc, rebuilding 4.9 million documents settled at 585 MB where a fresh process needs 229, because freed pages stayed in per-thread arenas. jemalloc with a one-second decay settled at 245 MB and improved writes by 22%. An earlier allocator trial had been rejected on a comparison against glibc's understated figure: the measurement was wrong, not the allocator.
- Never run a minute-long job inline in a control loop. Snapshot packing originally ran in the sealer loop, so for up to 130 seconds nothing uploaded, the memory guard never executed and clients timed out. Moving it to its own task at reduced priority turned 213,000 client timeouts into zero.
- Read the storage library's worker model before trusting its limits. The embedded engine sized its worker pool at one thread on this machine and ran compactions on it, so a long compaction blocked every memtable rotation and the write buffer grew without bound at replay speed. That was the cause of every rebuild failure in the soaks.
- Top-k pruning in the search index has preconditions. It only applies to a bare term query or a pure union of terms; wrapping the text clause in a boolean with one required clause silently scored every match. Peeling single-clause booleans gave 30% more reads.
Filter caches were worth building twice: evaluating an equality filter on a fast column rather than intersecting postings was worth 34%, and then caching a per-segment bitset applied inside the query, so that pruning stays active, was worth a further 61% on top.
And a small one with a large effect: a zero-copy slice over a store block, once cached, pins the whole 4 KB block in memory. A hundred and fifty thousand of those blew the memory gate. Copying the bytes out halved the cache footprint.
4What it does under load, and what it costs
A ten-minute soak wrote 14 million documents with zero errors. A rebuild from a 4.5 GB snapshot plus tail takes 40 seconds. On a real free-tier machine with real buckets in two regions it serves several thousand operations a second in bursts, for a measured cost of about a cent a day.
One application runs entirely on it, with no Postgres and no managed search behind it. That was the point of building it.
It is open source under the MIT licence, at github.com/madhudream/hanudb — the engine, the benchmark, the correctness suite, the deployment scripts, and results.tsv with all 108 experiments, the 27 discarded ones included. The discarded rows are the half I would want if I were reading somebody else's database, so they ship with it.
5What is still open
Single writer. The write ceiling is one core and one disk's fsync rate. Atomic multi-document batches exist; interactive transactions across requests do not. No secondary indexes beyond the dynamic keyword and fast columns, no query planner, no sharding. A hot set larger than memory falls back to the disk's random-read rate. Automatic failover needs a second machine, which is not free.
The harness core is faster than the free-tier machine's, so every harness figure here is an upper bound; the free-tier measurements are what an application should plan against. And the scored benchmark is a single insert-heavy workload — the update-heavy soak exists precisely because the benchmark does not exercise deletion, which means 108 experiments of tuning were all aimed at one shape of load.
6Evidence
The full technical account is written up as a preprint: HanuDB: A Cloud-Durable Document Database for Tiny Compute (23 pages at a paper's density).
Every step was written down at the time as a claim, its evidence and a verdict — HanuDB, tuned while I slept (6 parts). The discarded runs are in there too.
Where the work lives:
~/apps/hanudb— the engine, the loop, the results.~/apps/hanudb/paper— the preprint.~/apps/hanudb/results.tsv— one line per experiment.- Source: github.com/madhudream/hanudb — open source, MIT licence.
§Related writing and lessons
§To remember
9 recall cards are drawn from this thread. They come back on a schedule in §7 Recall. The first: What does the machine serving HanuDB actually hold?
← §2.5 Answers with the slide attached§2.7 Public data, kept honest →