Production-deployed durable write-ahead buffer in Rust — CRC32 checksums, configurable fsync tiers, automatic crash recovery via WAB segment replay, and a MySQL bulk-drain sink that compresses N records into a single server-side commit.
STACKRust · Write-Ahead Log · Durability · Storage Systems · MySQL · Postgres · Unix Sockets
HOW IT WORKS
Producers write records over a Unix domain socket; the daemon validates, CRC32-checksums, fsyncs, ACKs, and asynchronously drains batches to a pluggable sink.
WAB segments not reclaimed until the sink confirms the batch — on restart, unconfirmed segments are replayed automatically.
493/493 records recovered after kill -9 crash recovery test.
Deployed to production at bjj.space.
DESIGN
Four built-in sinks: noop (soak-testing), http (transient/permanent classification), mysql (N records → 1 INSERT → 1 server commit), postgres (ON CONFLICT DO NOTHING, TLS opt-in).
Shard-based MPMC queue — concurrent producer support without a global lock.
WAB flusher panics supervised and respawned (capped at 10 attempts before shard goes offline).