One WAL, four data models: how cross-modality transactions actually work

A previous post — The tradeoffs we made to fit four engines in one — said the trick is a single write-ahead log. That sentence is correct, dramatically under-specified, and exactly the kind of thing storage engineers stop trusting after their third bad weekend. So this post is the long version. What the WAL actually contains, how a four-modality transaction lands in it, what the fsync math looks like when a vector batch and a 50 MB blob arrive in the same group commit window, and the four mitigations that keep the cold-blob latency tail from eating the KV path.

If you have never had to reason about a WAL, the short version is: it is the ordered byte stream that every durable change is written to before anything else is allowed to acknowledge the write. Crash recovery replays it. Replication reads it. Backup snapshots are aligned to positions in it. It is the single most load-bearing file in the engine. Putting four data models behind one of these is the design decision that everything else in RedDB falls out of.

What “one WAL” actually means

The temptation, when you have four engines, is to give each one its own log. Postgres has pg_wal, you give the vector engine vec_wal, S3 has its own internal journal, Redis has its AOF. Four logs, four fsync queues, four recovery procedures. It composes — until you want a transaction that spans two of them. Then you are either writing two-phase commit by hand or you are pretending the consistency story is fine and hoping nobody graphs the divergence.

A single WAL means there is exactly one ordered byte stream on disk, and every durable mutation to every modality is appended to it before the client sees an ack. A transaction is, physically, a contiguous run of records in that stream bracketed by a BEGIN and a COMMIT record sharing a transaction id. When the COMMIT record’s LSN is durable — meaning fsync returned on the WAL file up to at least that byte offset — the transaction is committed. All of its modality writes are committed atomically because they are all before the COMMIT in the same ordered file. There is no second log to wait on.

This is how a single update can touch a document, append a vector, bump a KV counter and replace a blob and either all four of those changes survive a crash or none of them do. The atomicity is a property of byte ordering on disk, not a protocol.

Record layout

Every WAL record has the same envelope. The body differs per modality, but the envelope does not.

+----------+------+--------+---------+----------+-----------------+--------+
|   LSN    | size |  type  |  txn_id | modality |     payload     |  CRC32 |
+----------+------+--------+---------+----------+-----------------+--------+
| 8 bytes  |  4   |   1    |    8    |    1     | size - 22 bytes |   4    |
+----------+------+--------+---------+----------+-----------------+--------+

LSN — Log Sequence Number, monotonic, assigned at append time. The recovery cursor, the replication cursor, the snapshot cursor are all LSNs.
size — total record bytes including this header. Lets the replayer skip a corrupted record and continue if the CRC fails.
type — BEGIN, COMMIT, ABORT, WRITE, CHECKPOINT, BLOB_REF, FENCE. The first three bracket transactions; WRITE carries actual mutations; CHECKPOINT records a fuzzy checkpoint LSN; BLOB_REF is the sidecar pointer described below; FENCE is a barrier the replay cursor will not cross until acknowledged by the modality engines (used for online schema changes).
txn_id — 64-bit transaction id. Every WRITE carries the id of the transaction it belongs to.
modality — DOC, VEC, KV, BLOB, or META (system records). The replayer dispatches to the right engine using this byte.
payload — modality-specific bytes. The four modalities encode their payload differently:
- DOC — collection id, document id, MVCC version, encoded value (MessagePack).
- VEC — collection id, vector id, dimensionality, raw f32 bytes, optional metadata blob. The HNSW graph mutation is not in the WAL. It is recomputed at replay from the vector id and the existing graph state. This is the single most important encoding decision and we discuss it below.
- KV — namespace, key bytes, value bytes (or a tombstone marker), TTL.
- BLOB — blob id, content hash, size, and a sidecar offset. The blob’s actual bytes are not in this record. See the sidecar section.
CRC32 — checksum over the entire envelope including the payload. A failed CRC during replay marks the end of the durable log; anything after is treated as torn.

A transaction touching all four modalities therefore produces, at minimum:

BEGIN(txn=42)
WRITE(txn=42, modality=DOC, ...)
WRITE(txn=42, modality=VEC, ...)
WRITE(txn=42, modality=KV,  ...)
WRITE(txn=42, modality=BLOB, blob_ref=offset_in_sidecar)
COMMIT(txn=42)

Six records, one ordered run, one fsync to make the COMMIT durable. The four modality engines apply their respective WRITE records at replay time, in order, from the same cursor.

The lane diagram

What goes into the WAL is one question. What happens to those records on the way to and from disk is another. Here is the path:

   doc writer       vec writer       KV writer        blob writer
        |                |                |                 |
        v                v                v                 v
  +----------+     +----------+     +----------+     +-------------+
  |  encode  |     |  encode  |     |  encode  |     |  encode +   |
  |  payload |     |  payload |     |  payload |     |  hash blob  |
  +----------+     +----------+     +----------+     +------+------+
        \                \                /                 |
         \                \              /                  |
          \                \            /            +------+------+
           v                v          v             | blob bytes  |
        +---------------------------------+          |  to sidecar |
        |  shared in-memory WAL ring      |          |  (separate  |
        |  buffer (per-CPU sharded,       |          |   fsync     |
        |  global LSN assignment)         |          |   queue)    |
        +-------------------+-------------+          +------+------+
                            |                              |
                            v                              |
                  +---------------------+                  |
                  |   group commit      |  <--- blob ref ---+
                  |   coalescer         |
                  +---------+-----------+
                            |
                            v
                  +---------------------+
                  |   fsync(WAL file)   |
                  +---------+-----------+
                            |
                            v
                     clients ack'd
                  (up to COMMIT LSN)

Three details in that picture matter:

Per-CPU ring shards. The in-memory ring is sharded by core to avoid the lock contention you get when sixteen writer threads all want to append at once. LSN assignment is the only globally-ordered step; payload encoding and CRC happen on the writer’s local shard before LSN handoff.
The group commit coalescer. Many transactions’ COMMIT records land in the same fsync. A single fsync becomes a single durability event for every transaction whose COMMIT LSN is at or below the flushed byte offset. This is the standard Postgres / InnoDB trick and it is what keeps small-transaction throughput sane on slow disks.
The blob sidecar. Blob bytes never traverse the WAL ring. They go through a separate fsync queue with a different priority. The WAL gets a tiny BLOB_REF record that just points to where the bytes landed in the sidecar. We explain why below; it is the single biggest reason the cold-blob latency tail did not eat us.

Why the vector graph is not in the WAL

If you read the record layout carefully you will notice that a vector WRITE carries the raw f32 bytes but no HNSW graph mutation. That is deliberate and it took us two redesigns to land on.

An HNSW insert produces, in the worst case, mutations to several thousand graph edges across multiple layers. If we logged every edge mutation, the WAL bandwidth from a single 1024-dim vector insert would be 50–100× the bandwidth of the vector itself. Replay would be I/O bound on edge mutations instead of the engine’s own batched graph build.

So the WAL stores only the input — the vector and its id — and the graph is treated as derived state. At replay time, the vector engine re-runs the insert against the graph it has in memory. This makes replay slower per vector than a “log everything” approach would be, but it makes the steady-state WAL write rate dominated by the vectors themselves, which is the bound you actually want.

The cost: graph state is not point-in-time recoverable from the WAL alone. It is reconstructable from the WAL plus the most recent vector engine checkpoint. We checkpoint vector engines more aggressively than the other modalities (every five minutes by default, configurable) precisely because the replay cost is higher per record.

The same trick — “log the input, recompute the derived state” — is why we do not log internal LSM compaction events for the document engine, only the logical writes. Document compaction state is rebuilt from checkpoint + WAL replay.

The fsync contention tradeoff

Now the part that bites. Every modality’s COMMIT waits on the same fsync. A fsync is not free. On modern NVMe with the write cache flushed correctly (which you must, otherwise your “durable” log is a polite fiction), a single fsync costs roughly 50–200 microseconds in the steady state, climbing to single-digit milliseconds when the device’s internal cache is dirty and being drained.

Group commit amortises this for many concurrent transactions, so per-transaction fsync cost is much lower than the raw call cost. But the ceiling on transactions per second is bounded by fsync rate times group size. If your group size collapses — because writers are not arriving fast enough to coalesce, or because the records are huge and the group flush is slow — your tail latency grows.

The pathological case is the one the tradeoffs post named: a burst of blob writes lands in the same window as a steady stream of KV writes, and the KV transactions’ COMMIT is stuck waiting for the blob transactions’ bytes to flush. That is what produced the 1.8 ms → 14 ms P99 jump we measured under load. The flush window grew because the bytes in the window grew, and small transactions paid the price for being in the same group as large ones.

We are not going to pretend a shared WAL has no contention. It does. The four mitigations below are how we keep the contention from being your problem in the workloads we are appropriate for. Each one is a deliberate cost.

Mitigation 1 — Blob sidecar with pointer-only WAL records

The biggest single win. Blob bytes are not in the WAL. The WAL gets a BLOB_REF record — 64 bytes including the envelope — that points to an offset and length in a separate blob sidecar file. The sidecar has its own fsync queue, its own group commit coalescer, and crucially its own priority relative to the WAL.

The atomicity contract still works because we order the fsyncs: the blob sidecar fsync for a transaction must return before the WAL fsync that carries that transaction’s COMMIT. So at recovery time, if you see a COMMIT in the WAL, the bytes it references are guaranteed to be on disk in the sidecar. If the blob fsync succeeded but the WAL COMMIT fsync did not, the transaction is rolled back and the blob bytes are orphaned, to be reclaimed by the sidecar’s GC pass. Orphan blobs are cheap to detect (sidecar offsets with no live WAL reference) and reclaim is bounded by checkpoint cadence.

The cost: every blob write costs two fsyncs instead of one. For a workload dominated by tiny KV writes with occasional large blobs, that is a great trade — small writes are no longer queued behind large ones, and the second fsync for the blob is absorbed by a dedicated thread that the KV path never touches. For a workload of nothing but small blobs, it is a worse trade than putting them inline, and we set the inline-vs-sidecar threshold at 32 KB by default for that reason. Below 32 KB, blobs go through the WAL like everyone else.

Mitigation 2 — Group commit with adaptive batching

Group commit is not novel. The tuning is the part that matters.

The coalescer has two thresholds: a time deadline (default 500 µs) and a byte budget (default 1 MB). A flush fires when either threshold is hit. On a cluster with high small-transaction throughput, the byte budget is what bounds latency — you accumulate 1 MB of COMMIT records in well under 500 µs and flush. On a quiet cluster, the time deadline kicks in and a lone transaction waits at most 500 µs before its COMMIT is durable.

What we added on top is a feedback loop on the deadline. If the previous flush took longer than the deadline to complete (because the disk was busy), the next deadline is doubled, up to a ceiling of 4 ms. If flushes are consistently completing well under the deadline, it is halved, down to a floor of 100 µs. This keeps the flush rate matched to what the disk can actually sustain, which prevents a death-spiral where each flush starts before the previous one finishes and the queue grows without bound.

You can disable adaptive batching with wal_group_commit_adaptive = false if you want predictable behaviour for benchmarks. Most operators leave it on.

Mitigation 3 — `wal_priority` hint on the write path

Some writes are happy to commit a little later if it means small writes commit on time. Background ingestion, log-style audit writes, blob commits where the application has already shown the user “uploading…” — none of these need sub-millisecond commit latency.

The write API takes an optional wal_priority hint:

await db.transaction(async (tx) => {
  await tx.docs.put("conversations", convId, conversation)
  await tx.kv.set("counters/messages", count)
  await tx.blobs.put("attachments", attachmentId, blobBytes, {
    wal_priority: "background",  // <-- hint
  })
})

A background priority transaction’s COMMIT is allowed to coalesce into the next group commit window instead of the current one. In practice that adds anywhere from 100 µs to 4 ms of latency to the background transaction, and it removes that transaction’s bytes from the group that the foreground transactions are queued behind. The application still sees an atomic transaction; the fsync lane is just different.

The hint is opt-in. We do not auto-classify writes as foreground or background — the application knows which user-visible latency budget the write is on, and we are not going to guess.

Mitigation 4 — Per-tenant WAL bandwidth quotas

In a multi-tenant deployment, a single noisy tenant can starve every other tenant by saturating the shared WAL. We measured this once on staging — a single customer’s bulk import drove P99 KV latency for every other tenant on the node to 80 ms — and it convinced us to ship fair-share quotas before the next customer hit it in production.

The quota is enforced at the WAL ring entry point. Each tenant has a configurable byte-per-second budget (defaults derived from the plan tier) and a burst allowance. When a tenant exceeds budget, their writes are queued in a per-tenant overflow ring that drains into the shared WAL only when shared bandwidth is available. Other tenants are not blocked by an overflowing tenant’s queue depth.

The scheduling is weighted round-robin across tenant rings at the shared-ring entry, not strict priority, so a fully-loaded shared WAL still gives every active tenant proportional throughput rather than starving small tenants entirely. This is the model we mean when we say “fair-share” in the customer docs; the previous round-robin-only scheduler did starve small tenants under load, and we changed it after the same staging incident.

What this gets you

The point of all this machinery is that the application layer can write the obvious code:

await db.transaction(async (tx) => {
  // Document write
  const conv = await tx.docs.put("conversations", convId, {
    title, participants, updated_at: now(),
  })
  // Vector insert (chat embedding)
  await tx.vecs.upsert("conversations.embeddings", convId, embedding)
  // KV counter
  await tx.kv.incr(`counters/messages/${userId}`)
  // Blob commit (attachment)
  await tx.blobs.put("attachments", attachmentId, fileBytes)
})

…and have it be a single transaction. If the process dies between any two of those lines, none of them are visible. If it dies after the COMMIT returns, all of them are. There is no two-phase commit between four engines. There is no compensating write to roll back the vector if the blob upload fails. There is one ordered log, one fsync, one cursor.

This is the thing the stitched stack — Postgres + pgvector + S3 + Redis — cannot give you. You can come close, with careful application-layer patterns and idempotency keys everywhere, and a lot of teams do. But the consistency you get is the consistency you write, not the consistency the storage gives you. The thing we are selling is the inversion of that.

When you should still pick a stitched stack

The shared WAL is the feature and the cost. It is the right architecture for workloads where the seams between modalities are where the bugs live. It is the wrong architecture if the seams are not where your bugs live and your single-modality tuning headroom matters more.

Concretely, do not buy us if:

Your workload is dominated by 100+ MB blob writes at high concurrency and you also need sub-5 ms reads on the same cluster. The blob sidecar mitigates this but does not eliminate it; the disk is still shared.
You need to tune a vector workload to recall-at-99.5% on an unusual embedding model and you know which HNSW knobs you would change. We do not expose them per-collection.
Your isolation requirements are strict serializable and the write-skew workarounds we already documented are not acceptable to your domain.

For everyone else — which in our experience is most teams shipping AI-adjacent applications where the data crosses modalities on every request — the shared WAL is the boring choice. We meant it to be boring. Storage that surprises you is storage you cannot operate.

What lives in this post and what does not

This post is the WAL. Not the rest of the engine. If you want the surrounding context, three companion posts cover the pieces this one assumes:

The tradeoffs we made to fit four engines in one — the bill for sharing a WAL, in language a buyer reads before signing.
Snapshot isolation, in practice — what the consistency model on top of this WAL actually gives an application, and what it does not.
Stop stitching Postgres, pgvector, S3, and Redis — the case for why a shared WAL is worth its bill in the first place.

The deeper specs — the recovery state machine, the replication protocol that reads from the same LSN cursor, the snapshot-and-restore protocol that aligns to checkpoint LSNs — will get their own posts. They all build on the record layout above. If you only remember one diagram from this post, make it the lane diagram. Everything else is implementation.