Weir -- Durable Write Buffer

weir sits between a high-rate producer and a slow or flaky downstream. It acks from a local CRC32-checksummed write-ahead buffer (WAB), then drains to the sink in bulk, reclaiming segments only after the sink confirms them. It is deliberately not a broker (one stream, one sink), not embedded (a daemon), and not a database (the buffer isn't queryable). Delivery is at-least-once, so sinks dedupe.

• Producers push over a Unix domain socket (or TCP + mutual TLS); the daemon validates the frame, appends to the WAB, fsyncs per the requested tier, acks, and drains batches to a Sink asynchronously. Unconfirmed segments replay on restart.
• Ten-crate workspace: weir-core (wire types), weir-wab (segment format + reader), weir-server (Unix-only daemon), weir-client, weir-sink-sdk (Sink trait), weir-sink-s3, weir-attest, weir-ctl (admin CLI), weir-rs (facade re-exporting the libraries), weir-testkit.
• Six built-in sinks: noop, http, mysql, postgres by default; clickhouse and s3 (AWS S3, MinIO, Cloudflare R2, Backblaze B2, Ceph) behind opt-in features. The S3 sink is built against weir-sink-sdk alone, proving the SDK is a usable third-party contract.
• WAB flusher and drain threads are panic-supervised; dead-letter segments can be listed, dropped or requeued with weir-ctl dl.

• Two tiers, chosen per record: Durable fsyncs before the ack (fdatasync on Linux); Buffered acks before the fsync and survives a process crash, not power loss.
• Measured under simulated power loss (kill -9 + dm-flakey dropped writes + remount, on one Linux host, uncompressed WAB): 24 h, 5,311 episodes, 42,814,591 acked Durable records, none lost. In a separate 6 h Buffered run (706 episodes), loss was bounded by one writeback interval: worst case 1.76 s uncompressed (~5.7 s with zstd).
• On macOS weir uses F_BARRIERFSYNC, which orders writes but is not guaranteed across power loss; Linux is the platform the guarantee is claimed for.

• 2.0: Sync and Batched merged into one Durable tier (they had carried the same guarantee since both moved to batch-boundary group fsync) and the Sink trait changed shape. The retired 0x02 tier byte still decodes, so 1.x producers keep working unchanged.
• 2.1: S3 object-storage sink as its own crate.
• 3.0: PushTracked/AckTracked -- push_tracked returns a RecordCoordinate (segment, index, RecordId) so a producer knows which record it wrote; plus wab_segment_max_lifetime_secs, a seal deadline a steady trickle can't postpone.
• 4.0: wire-level batching -- PushBatch/AckBatch carry up to 2,048 records per frame, answered with an LSB-first positional bitmap; MessageType became #[non_exhaustive] so future message types stay additive.
• 4.0: weir-attest -- a SHA-256 hash chain over sealed segments, driven by weir-ctl attest seal|verify|head. It detects after-the-fact edits to a sealed buffer once the chain head is shipped off-host; it doesn't prove authorship or prevent writes.
• Each Rust API break was a major version; WIRE_VERSION has stayed 1 throughout.

• Deterministic simulation testing checks WAB durability invariants under injected crash/fault schedules with replayable seeds -- a 300-seed sweep on every CI run.
• Language-neutral conformance vectors: 30 frozen v1 wire vectors, 26 batch, 11 tracked-push and 5 attest-chain vectors. C, Go, Java, Python and TypeScript demo clients implement the spec independently rather than porting the Rust code.
• 70 system tests that spawn the real daemon (crash/restart, replay, drain), cargo-fuzz targets on the trust-boundary parsers, and kill -9 and power-cut chaos soaks.
• Research write-ups in docs/benchmarks/: a bare-metal capture with a measured noise floor; sustained load (one Durable producer on an i9-9900K + ext4 on a SATA SSD: 877 rec/s short-run vs 352 rec/s averaged over two multi-hour runs, traced to the SSD's fsync oscillation rather than weir); buffering and recovery (WAB growth plateaus and backpressure reaches push); and when WAB zstd compression pays off by record size.
• The README quotes no latency headline: CI runs on shared 2-vCPU runners, so any figure it gives cites named hardware and its fsync primitive.