RedDB blog

RedDB on a Raspberry Pi: the smallest sane install

2026-06-09

The pitch we keep making is that RedDB collapses four engines into one — document, vector, KV, blob — sharing one WAL, one auth surface, one ordered cursor for change capture. That story sounds different on a server with 64 cores and 512 GB of RAM than it does on a Raspberry Pi 5 that draws less power than the desk lamp it sits next to.

So we put one on a Pi 5 for a weekend. This is the playbook and the numbers.

What you need

Raspberry Pi 5, 8 GB model. The 4 GB will technically run but the HNSW index for the demo dataset spills under memory pressure and your numbers will be noise.
A real SSD over USB 3, or an NVMe HAT. Do not run this on the microSD card. RedDB's WAL is fsync-heavy; an SD card will both bottleneck writes and wear out in a month.
The active cooler. The CPU will sit at 65–70 °C under load and will throttle without it.

Total cost of the rig as we built it: roughly $130 for the Pi + cooler + power supply + 256 GB NVMe + HAT. Less than a single month of the smallest managed-Postgres tier on any cloud.

Image and install

We used the 64-bit Pi OS Lite release, no desktop. Flash it with the official imager, enable SSH from the imager's settings pane before writing, and boot.

Then:

ssh pi@reddb-pi.local
sudo apt update && sudo apt install -y curl
curl -fsSL https://get.reddb.io/install.sh | sh

The install script detects aarch64, pulls the ARM64 build, drops a reddb binary in /usr/local/bin, and writes a systemd unit at /etc/systemd/system/reddb.service pointing the data directory at /var/lib/reddb. If you mounted the NVMe at /mnt/nvme, point the data dir there instead before starting:

sudo sed -i 's|/var/lib/reddb|/mnt/nvme/reddb|' /etc/systemd/system/reddb.service
sudo systemctl daemon-reload
sudo systemctl enable --now reddb

Health check:

curl http://localhost:7878/health
# {"status":"ok","version":"...","uptime_ms":1843}

That is the whole install. There is no second engine to provision, no extension to compile, no sidecar to wire up.

Load a dataset

We used a tiny slice of an open product catalog — 100,000 rows, each with a title, a 256-character description, and a 384-dimensional embedding generated locally with all-MiniLM-L6-v2 (which itself runs on the Pi, slowly but it runs).

Create the table and the index:

CREATE TABLE products (
  id          UUID PRIMARY KEY,
  title       TEXT NOT NULL,
  description TEXT NOT NULL,
  price_cents INT NOT NULL,
  embedding   VECTOR(384) NOT NULL
);

CREATE INDEX products_embedding_idx
  ON products
  USING hnsw (embedding vector_cosine_ops);

Bulk-load over the line-delimited JSON import path:

reddb import --table products --format ndjson < products.ndjson

Load time for 100,000 rows on NVMe: 94 seconds, including HNSW index construction with default M=16 and ef_construction=200. That works out to roughly 1,064 rows per second, embedding bytes included.

Run a query

Pick a vector to search with — in practice the embedding of a user's query, here we just pulled one from a random row to keep the demo self-contained — and run nearest-neighbor search filtered by price:

SELECT id, title, price_cents,
       1 - (embedding <=> $1) AS similarity
FROM products
WHERE price_cents < 5000
ORDER BY embedding <=> $1
LIMIT 10;

Latency numbers, warm cache, single client, ef_search=64:

Operation	P50	P99
KV `GET` (small value)	0.4 ms	1.1 ms
Document point read by primary key	0.6 ms	1.4 ms
Vector top-10 over 100k rows, no filter	8 ms	18 ms
Vector top-10 with `price_cents` predicate	11 ms	24 ms
Single-row upsert (text + vector, one tx)	4 ms	9 ms
10 KB blob write + immediate read	6 ms	14 ms

None of these are competitive with a fat server. All of them are well inside what makes for a responsive demo, a homelab assistant, an on-device search UI, or a workshop you give to a room full of students who can each have their own Pi.

What this is and is not

This is portability proof. The same binary, the same protocol, the same SQL surface that runs on a c7g.8xlarge runs on $130 of hardware on your desk. There is no "lite" build, no missing modality, no separate config flag for "small mode."

It is not a production deployment target for a real customer-facing system. The Pi's single SSD is your single point of failure, the network is whatever's in your house, and one CPU throttle event takes you offline. Treat it as a hobby rig, a dev environment, a teaching tool, an edge node where the alternative is no database at all.

What we like about this install is what it implies about every other install. If RedDB fits on a Pi, then the operator complexity of the four-engine stack we replaced — Postgres, Qdrant, Redis, S3, each with its own backups, auth, monitoring, version — was never inherent to the problem. It was the toll of the stitching.

The smallest sane install is one engine, one binary, one systemd unit. Everywhere else, you just need a bigger Pi.

Cross-session task queues — AFK agents with durable resume

2026-06-07

The AFK promise that breaks at 3am

You hand the agent a 40-item backlog before bed. Eight hours later you wake up to find it crashed on item 12, the context window got compacted past the point of useful, and the remaining 28 items live only inside a transcript nobody will read. Or worse — it finished items 1–11, started 12, crashed, restarted, and ran 12 again (writing duplicate rows, double-posting the PR comment, double-charging the API).

The Ralph loop and every other AFK pattern share the same failure mode: the task list lives in the agent's head. Stop the agent and the list vanishes. Restart it and there's no idempotency on what was already done.

The fix is unromantic: model the task list as rows in RedDB. State machine on each row. Idempotency key on each side-effect. Watchdog that reclaims expired leases. Sibling to D12's sub-agent dispatch queue but pointed inward — the agent's own work, not its fan-out.

The schema: state, checkpoint, idem_key

CREATE TYPE afk_state AS ENUM ('todo', 'running', 'paused', 'done', 'failed');

CREATE TABLE afk_task (
  id            uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  agent_id      text NOT NULL,                      -- which AFK runner owns this list
  ordinal       int  NOT NULL,                      -- stable order in the backlog
  title         text NOT NULL,
  spec          jsonb NOT NULL,                     -- prompt + acceptance criteria
  state         afk_state NOT NULL DEFAULT 'todo',
  checkpoint    jsonb,                              -- mid-task progress: files touched, sub-results
  idem_key      text UNIQUE,                        -- one row per (agent, logical-task)
  lease_until   timestamptz,                        -- null when not running
  attempts      int NOT NULL DEFAULT 0,
  max_attempts  int NOT NULL DEFAULT 5,
  last_error    text,
  created_at    timestamptz NOT NULL DEFAULT now(),
  updated_at    timestamptz NOT NULL DEFAULT now()
);

CREATE INDEX afk_task_next_idx
  ON afk_task (agent_id, ordinal)
  WHERE state IN ('todo','paused');

CREATE INDEX afk_task_lease_idx
  ON afk_task (lease_until)
  WHERE state = 'running';

Three columns do the heavy lifting:

state — the row's place in the lifecycle. todo → running → done on the happy path; running → paused on a clean Stop; running → failed after max_attempts; paused → running on resume.
checkpoint — every non-trivial step writes its progress here. Files modified, sub-prompts completed, partial scores. The agent that picks the task up after a crash reads this before deciding what to do next.
lease_until — set when a worker claims the row, refreshed on each checkpoint. The watchdog reclaims rows whose lease expired (worker crashed without releasing).

Enqueue: idempotent and ordered

The dispatcher is whatever produces the backlog — a /plan command, a to-issues skill output, a CI job. It writes rows with ON CONFLICT DO NOTHING on idem_key:

import { Client } from 'pg'

export async function enqueue(
  pg: Client,
  agentId: string,
  tasks: { title: string; spec: object; idemKey: string }[],
) {
  await pg.query('BEGIN')
  try {
    const startOrdinal = (
      await pg.query(
        `SELECT COALESCE(MAX(ordinal), 0) AS m FROM afk_task WHERE agent_id = $1`,
        [agentId],
      )
    ).rows[0].m

    for (const [i, t] of tasks.entries()) {
      await pg.query(
        `INSERT INTO afk_task (agent_id, ordinal, title, spec, idem_key)
         VALUES ($1, $2, $3, $4, $5)
         ON CONFLICT (idem_key) DO NOTHING`,
        [agentId, startOrdinal + i + 1, t.title, t.spec, t.idemKey],
      )
    }
    await pg.query('COMMIT')
  } catch (e) {
    await pg.query('ROLLBACK')
    throw e
  }
}

idem_key is the contract. "afk:ralph:issue-145:v1" survives every restart and every duplicate dispatch. Re-running the same /plan is a noop.

Claim: SKIP LOCKED + lease

The agent's run loop claims one row at a time:

const CLAIM = `
  WITH next AS (
    SELECT id FROM afk_task
     WHERE agent_id = $1
       AND state IN ('todo','paused')
     ORDER BY ordinal
     FOR UPDATE SKIP LOCKED
     LIMIT 1
  )
  UPDATE afk_task t
     SET state       = 'running',
         lease_until = now() + interval '15 minutes',
         attempts    = t.attempts + 1,
         updated_at  = now()
    FROM next
   WHERE t.id = next.id
  RETURNING t.*;
`

export async function claimNext(pg: Client, agentId: string) {
  const r = await pg.query(CLAIM, [agentId])
  return r.rows[0] ?? null
}

FOR UPDATE SKIP LOCKED is what makes this safe under multiple parallel runners on the same agent_id — two machines running the same AFK loop will never grab the same row. 15-minute lease is a starting point; tune to the longest reasonable task duration.

Checkpoint: write often, write cheap

Every meaningful step inside the task body writes a checkpoint. The frequency is the recovery-cost knob — too sparse and a crash redoes hours of work; too dense and you bloat the row.

export async function checkpoint(
  pg: Client,
  taskId: string,
  patch: object,
) {
  await pg.query(
    `UPDATE afk_task
        SET checkpoint = COALESCE(checkpoint, '{}'::jsonb) || $2::jsonb,
            lease_until = now() + interval '15 minutes',
            updated_at  = now()
      WHERE id = $1`,
    [taskId, patch],
  )
}

The jsonb || jsonb merge keeps it additive — no read-modify-write race between the worker and a stop-handler. Inside a Ralph iteration the natural checkpoint boundaries are: after exploration, after first edit, after pnpm test passes, after commit. Four writes per task is plenty.

Resume: read the checkpoint, don't redo it

On startup the agent reads its own state before claiming a new row:

const r = await pg.query(
  `SELECT * FROM afk_task
    WHERE agent_id = $1 AND state = 'running'
    ORDER BY ordinal LIMIT 1`,
  [agentId],
)
if (r.rows[0]) {
  // crashed mid-flight; the row's lease will expire and the watchdog
  // will flip it to 'paused' for us. Pick it up explicitly:
  return resumeFromCheckpoint(r.rows[0])
}

resumeFromCheckpoint is the per-task-kind logic — for a Ralph iteration that means re-reading the issue file plus the checkpoint's files_touched list, skipping the parts already done, picking up at the next unmet acceptance criterion. The checkpoint isn't a magic resume — it's a structured note from past-self to future-self.

Watchdog: reclaim, retry, give up

A separate process (cron, systemd timer, or a Stop hook wired into the agent itself) runs every minute:

-- 1. Reclaim expired leases
UPDATE afk_task
   SET state = CASE
                 WHEN attempts >= max_attempts THEN 'failed'
                 ELSE 'paused'
               END,
       lease_until = NULL,
       last_error  = COALESCE(last_error, 'lease expired'),
       updated_at  = now()
 WHERE state = 'running'
   AND lease_until < now();

-- 2. Surface poison pills
SELECT id, agent_id, title, attempts, last_error
  FROM afk_task
 WHERE state = 'failed'
   AND updated_at > now() - interval '1 hour';

The first statement is the resume mechanism — a crashed worker's row gets flipped back to paused so the next claim picks it up. The second is the dead-letter view that the human (or an orchestrator) checks in the morning.

Stop handler: graceful pause, not a crash

Claude Code emits a Stop event when the user cancels mid-flight. Wiring a hook to it flips the row cleanly:

{
  "hooks": {
    "Stop": [
      {
        "matcher": ".*",
        "command": "psql $RDB_URL -c \"UPDATE afk_task SET state='paused', lease_until=NULL, updated_at=now() WHERE agent_id='$AFK_AGENT_ID' AND state='running'\""
      }
    ]
  }
}

The difference between paused and a watchdog-reclaimed running is intent — a paused row was put down deliberately; a running row past its lease was abandoned. The watchdog treats them identically on resume, but the audit trail keeps the distinction.

When to skip this

Three cases where adding a queue is overkill:

Single-task agents. If the agent does one thing and exits, the durability surface is the commit, not the queue. Don't wrap a /fix-one-bug flow in this.
Sub-second tasks. Lease overhead dominates. Use the in-process Task tool; the queue exists for tasks measured in minutes, not seconds.
Tasks with no idempotency story. If you can't write a stable idem_key, you can't safely retry. Either fix the side-effect to be idempotent or accept the risk and document it.

Tying it together

The Ralph loop and every AFK pattern in this series — /remember commands, eval replay, sub-agent fan-out — all become noticeably more boring once their state machine is a table you can SELECT from. The observability queries from D9 join straight against afk_task so "what's my agent doing right now?" stops being a transcript-grep problem and starts being a SQL problem. Boring is the goal.

Eval datasets in RedDB — from ad-hoc prompts to versioned regression sets

2026-06-05

The notebook eval is a dead end

Pick any team running an agent in production for more than a quarter and look at how they evaluate it. You will find a Jupyter notebook, an evals.py, or a folder of .txt prompts. Someone wrote them the week the agent shipped. Nobody has re-run them in two months. When a new model drops — Claude Opus 4.7 last month, Sonnet 4.6 the week before — the same notebook gets opened, a few cells get edited, results are eyeballed, "looks fine," done.

That is not an eval. That is a memory of an eval.

The problems compound:

No baseline. The output of last month's run is gone. You cannot diff.
No coverage signal. Which production scenarios are covered? No one knows. The prompts that matter — the ones that broke last quarter — are not in the file.
No tagging. "Show me the long-context cases" requires grep over a folder.
No fresh data. Every interesting failure the agent had this sprint is in a Slack thread, not in the eval set.

The fix is the one a database engineer would write on a napkin: every interesting interaction is a row, tagging is a column, versioning is a dataset_version table, replay is a SELECT … WHERE tag = 'long-context', and the model upgrade is a JOIN between two run tables. Builds on the memory schema from D1, the observability tables from D9, and the dispatch queue from D12 (workers replay the set in parallel).

The schema: cases, datasets, runs

Three tables. Cases are the facts. Datasets are versioned snapshots. Runs are the result of replaying a dataset against one model + prompt revision.

-- A single captured interaction. Append-only.
CREATE TABLE eval_case (
  id              uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  source          text NOT NULL,                  -- 'production' | 'incident' | 'handcrafted'
  prompt          text NOT NULL,                  -- full user turn, including system context
  context         jsonb NOT NULL DEFAULT '{}',    -- files, tool history, env
  reference       text,                           -- gold answer if known; NULL = judged at run time
  tags            text[] NOT NULL DEFAULT '{}',   -- ['long-context','tool-use','regression-#1284']
  captured_at     timestamptz NOT NULL DEFAULT now(),
  captured_by     text NOT NULL,                  -- agent id or user
  embedding       vector(1024)                    -- voyage-3, for "find similar cases"
);

CREATE INDEX eval_case_tags_idx  ON eval_case USING gin (tags);
CREATE INDEX eval_case_embed_idx ON eval_case USING hnsw (embedding vector_cosine_ops);

-- A frozen, named snapshot of cases. Immutable once published.
CREATE TABLE eval_dataset (
  id          uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  name        text NOT NULL,                      -- 'core-v3', 'long-context-2026-q2'
  version     int  NOT NULL,
  case_ids    uuid[] NOT NULL,                    -- locked at publish
  notes       text,
  published_at timestamptz NOT NULL DEFAULT now(),
  UNIQUE (name, version)
);

-- One row per (dataset_version, case, model+prompt revision) replay.
CREATE TABLE eval_run (
  id            uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  dataset_id    uuid NOT NULL REFERENCES eval_dataset(id),
  case_id       uuid NOT NULL REFERENCES eval_case(id),
  model         text NOT NULL,                    -- 'claude-opus-4-7'
  prompt_rev    text NOT NULL,                    -- git sha of the system prompt
  output        text NOT NULL,
  pass          boolean,                          -- NULL if judged later
  judge_score   numeric(3,2),                     -- 0.00–1.00
  judge_reason  text,
  latency_ms    int  NOT NULL,
  input_tokens  int  NOT NULL,
  output_tokens int  NOT NULL,
  cost_usd      numeric(10,6) NOT NULL,
  ran_at        timestamptz NOT NULL DEFAULT now()
);

CREATE INDEX eval_run_dataset_idx ON eval_run (dataset_id, model, prompt_rev);

Three design choices to call out:

Cases are append-only. A captured case never mutates. If the gold answer was wrong, you write a new case and tag the old one superseded. This is how you get clean diffs across versions.
Datasets freeze case_ids. Publishing a dataset is a snapshot. Six months from now you can rerun core-v3 and the membership has not drifted.
prompt_rev is the git SHA of the system prompt. Without it, a regression looks like a model regression when it is actually a prompt edit. Always log both.

Capture: the hook that fills the table

Capturing has to be cheap or nobody does it. A PostToolUse hook with a slash-command escape hatch covers both flows:

Auto-capture any turn that ends with an explicit user thumbs-down (/eval-bad) or a hook-detected failure (timeout, retry-exhausted, judge score < threshold).
Manual capture via /eval-capture <tag1,tag2> when the human notices something worth keeping.

/eval-capture is one shell script. It reads the current conversation tail, embeds the prompt with voyage-3, and INSERTs:

#!/usr/bin/env bash
# ~/.claude/commands/eval-capture.sh
set -euo pipefail

tags="${1:-handcrafted}"
prompt="$(claude-cli conversation tail --turns 1 --format text)"
context="$(claude-cli context dump --format json)"

embedding="$(curl -s https://api.voyageai.com/v1/embeddings \
  -H "Authorization: Bearer $VOYAGE_API_KEY" \
  -H 'Content-Type: application/json' \
  -d "$(jq -nc --arg i "$prompt" \
        '{model:"voyage-3", input:[$i]}')" \
  | jq -c '.data[0].embedding')"

psql "$REDDB_URL" -v ON_ERROR_STOP=1 <<SQL
INSERT INTO eval_case (source, prompt, context, tags, captured_by, embedding)
VALUES (
  'handcrafted',
  \$\$${prompt//\$\$/\$\$\$\$}\$\$,
  '${context}'::jsonb,
  string_to_array('${tags}', ',')::text[],
  '${USER}',
  '${embedding}'::vector
);
SQL

echo "captured to eval_case (tags: ${tags})"

PostToolUse is the same idea wrapped around a failure detector — drop in the transactional-hook pattern from D3 so a half-written embedding never lands without its row.

Publish: freezing a dataset

A dataset is a one-line query plus an insert:

-- Pin everything captured this quarter that is not superseded.
WITH picked AS (
  SELECT id FROM eval_case
  WHERE captured_at >= '2026-04-01'
    AND NOT ('superseded' = ANY(tags))
)
INSERT INTO eval_dataset (name, version, case_ids, notes)
SELECT 'core', 4,
       array_agg(id ORDER BY captured_at),
       'Q2 2026 baseline, includes the 1284 long-context regression'
FROM picked;

Membership is now frozen. Anyone — six months, three model upgrades, two prompt rewrites from now — can rerun core v4 and get the same case set.

Replay: the eval is a query

Replay is a function over (dataset_id, model, prompt_rev). The naive single-process version is fine for ~200 cases; for anything bigger, push each case into the dispatch queue from D12 and let N workers chew through it in parallel.

// scripts/replay.ts
import { Client } from 'pg'
import Anthropic from '@anthropic-ai/sdk'

const db  = new Client({ connectionString: process.env.REDDB_URL })
const ai  = new Anthropic()
const MODEL       = process.env.MODEL       ?? 'claude-opus-4-7'
const PROMPT_REV  = process.env.PROMPT_REV  ?? execSyncSha('prompts/system.md')
const DATASET     = process.env.DATASET     ?? 'core'
const VERSION     = Number(process.env.VERSION ?? 4)

await db.connect()

const { rows: [ds] } = await db.query(
  `SELECT id, case_ids FROM eval_dataset WHERE name=$1 AND version=$2`,
  [DATASET, VERSION],
)

const { rows: cases } = await db.query(
  `SELECT id, prompt, reference FROM eval_case WHERE id = ANY($1::uuid[])`,
  [ds.case_ids],
)

for (const c of cases) {
  const t0 = Date.now()
  const res = await ai.messages.create({
    model: MODEL,
    max_tokens: 2048,
    system: await fs.readFile('prompts/system.md', 'utf8'),
    messages: [{ role: 'user', content: c.prompt }],
  })
  const out = res.content.map(b => b.type === 'text' ? b.text : '').join('')

  const { pass, score, reason } = c.reference
    ? exactMatch(out, c.reference)
    : await llmJudge(out, c.prompt)        // separate small-model call

  await db.query(
    `INSERT INTO eval_run
       (dataset_id, case_id, model, prompt_rev, output,
        pass, judge_score, judge_reason, latency_ms,
        input_tokens, output_tokens, cost_usd)
     VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12)`,
    [ds.id, c.id, MODEL, PROMPT_REV, out,
     pass, score, reason, Date.now() - t0,
     res.usage.input_tokens, res.usage.output_tokens, priceUsd(res.usage, MODEL)],
  )
}

There is no eval framework here. There is a for loop and an INSERT. That is the whole point — the eval is the SQL you write afterwards.

The eval is the SQL afterwards

This is what you actually wanted all along.

-- Pass rate per model on the latest core dataset
SELECT model, prompt_rev,
       count(*)            AS n,
       avg(pass::int)::numeric(4,3) AS pass_rate,
       avg(judge_score)::numeric(4,3) AS mean_score,
       sum(cost_usd)::numeric(8,2)   AS run_cost
FROM eval_run r
JOIN eval_dataset d ON d.id = r.dataset_id
WHERE d.name = 'core' AND d.version = 4
GROUP BY model, prompt_rev
ORDER BY pass_rate DESC;

-- Regressions: cases the new model fails that the old one passed
WITH old AS (
  SELECT case_id, pass FROM eval_run
  WHERE model = 'claude-opus-4-6' AND prompt_rev = 'a1b2c3'
), new AS (
  SELECT case_id, pass, output FROM eval_run
  WHERE model = 'claude-opus-4-7' AND prompt_rev = 'a1b2c3'
)
SELECT c.id, c.prompt, c.tags, new.output
FROM eval_case c
JOIN old ON old.case_id = c.id
JOIN new ON new.case_id = c.id
WHERE old.pass AND NOT new.pass
ORDER BY 'regression' = ANY(c.tags) DESC;

-- Coverage by tag — which areas of the dataset are thin?
SELECT unnest(tags) AS tag, count(*) AS cases
FROM eval_case
WHERE id = ANY((SELECT case_ids FROM eval_dataset WHERE name='core' AND version=4))
GROUP BY 1
ORDER BY 2 DESC;

-- "Find me cases like the one that just broke production"
SELECT id, prompt, tags
FROM eval_case
ORDER BY embedding <=> (SELECT embedding FROM eval_case WHERE id = $1)
LIMIT 10;

That last one is the killer feature CLI evals do not have. When a production failure lands, you embed the failing prompt and the database hands you the ten most-similar historical cases — your regression set writes itself.

When this is overkill

Solo prototype, two prompts. Stay in a notebook. You will outgrow it; you have not yet.
No judge signal. If neither exact-match nor an LLM judge can score the output, the table is just a log. Fix the judging problem before the storage problem.
Daily replay against a frontier model. Cost discipline matters more than capture discipline. Sample first, run nightly on a small slice, full runs only on model bumps and prompt-revision PRs.

The shape we end up with — cases are facts, datasets are snapshots, runs are joins — is the same shape every mature evaluation system has, from ML research benchmarks to compiler test suites. The novelty is just that the agent feeds itself: every interesting turn it ever has becomes a row that the next model release has to answer for.

Sub-agent dispatch via database queue — durable parallelism across sessions

2026-06-03

The Task tool's shelf life is one session

Task(subagent_type=…) is the most underused tool in Claude Code. Spawn five sub-agents, each chews on an independent slice, parent collects results — except every one of those sub-agents lives inside the parent's process. Close the laptop, kill the terminal, hit a context-window cap on the parent, and the work evaporates. Mid-flight tool calls are simply gone; their partial output is unrecoverable.

That's fine for "explore the codebase, summarize" — disposable. It's not fine for "run the eval suite against 200 prompts," "lint and auto-fix 80 packages," "regenerate embeddings for every doc touched this week." Those are the jobs you actually want to send AFK. They need to survive a restart.

The pattern in this post: push sub-agent tasks into a RedDB queue, let one or more workers pull from it, write results back to the same row, and have the parent session subscribe to completions via LISTEN/NOTIFY. Same Task ergonomics on the surface, durable fan-out underneath. Builds on the memory backend from D1, the transactional-hook pattern from D3, and the observability schema from D9.

The schema: one table, three states, a notify trigger

CREATE TYPE task_state AS ENUM ('pending', 'running', 'done', 'failed');

CREATE TABLE agent_task (
  id           uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  parent_id    text NOT NULL,                       -- session id of dispatcher
  kind         text NOT NULL,                       -- 'review', 'embed', 'eval'…
  payload      jsonb NOT NULL,                      -- prompt, file list, cfg
  state        task_state NOT NULL DEFAULT 'pending',
  worker_id    text,
  attempts     int NOT NULL DEFAULT 0,
  max_attempts int NOT NULL DEFAULT 3,
  result       jsonb,
  error        text,
  idem_key     text UNIQUE,                         -- dedupe on retry
  created_at   timestamptz NOT NULL DEFAULT now(),
  started_at   timestamptz,
  finished_at  timestamptz,
  visible_at   timestamptz NOT NULL DEFAULT now()   -- delayed retry, lease
);

CREATE INDEX agent_task_claim_idx
  ON agent_task (visible_at)
  WHERE state = 'pending';

CREATE INDEX agent_task_parent_idx ON agent_task (parent_id, state);

CREATE OR REPLACE FUNCTION notify_task_done() RETURNS trigger AS $$
BEGIN
  IF NEW.state IN ('done','failed') AND OLD.state <> NEW.state THEN
    PERFORM pg_notify('task_done_' || NEW.parent_id, NEW.id::text);
  END IF;
  RETURN NEW;
END $$ LANGUAGE plpgsql;

CREATE TRIGGER agent_task_notify
  AFTER UPDATE ON agent_task
  FOR EACH ROW EXECUTE FUNCTION notify_task_done();

Four moving parts worth a comment. idem_key lets a flaky dispatcher retry without double-queuing — same key, second insert returns the existing row. visible_at doubles as both delayed retry (push it ten seconds into the future after a failure) and worker lease (a claimer bumps it 5 minutes out so a peer doesn't steal in-flight work). The partial index on (visible_at) WHERE state = 'pending' keeps the claim query a single index range scan even at a million rows. The pg_notify channel is per-parent so a dispatcher hears only its own completions — no fan-out storm to unrelated sessions.

Dispatcher: a tiny wrapper around Task

// .claude/lib/dispatch.ts
import { Pool } from 'pg'
const pool = new Pool({ connectionString: process.env.REDDB_URL })

export async function dispatch(kind: string, payload: unknown, opts: {
  parentId: string
  idemKey?: string
  maxAttempts?: number
}) {
  const { rows } = await pool.query(
    `INSERT INTO agent_task (parent_id, kind, payload, idem_key, max_attempts)
     VALUES ($1, $2, $3, $4, COALESCE($5, 3))
     ON CONFLICT (idem_key) DO UPDATE SET idem_key = EXCLUDED.idem_key
     RETURNING id`,
    [opts.parentId, kind, payload, opts.idemKey ?? null, opts.maxAttempts ?? null],
  )
  return rows[0].id as string
}

export async function awaitAll(parentId: string, ids: string[], timeoutMs = 600_000) {
  const client = await pool.connect()
  try {
    await client.query(`LISTEN "task_done_${parentId}"`)
    const pending = new Set(ids)
    const results = new Map<string, any>()

    const drain = async () => {
      const { rows } = await client.query(
        `SELECT id, state, result, error FROM agent_task
          WHERE parent_id = $1 AND id = ANY($2::uuid[]) AND state IN ('done','failed')`,
        [parentId, [...pending]],
      )
      for (const r of rows) { results.set(r.id, r); pending.delete(r.id) }
    }
    await drain()

    if (pending.size === 0) return results
    return await new Promise<typeof results>((resolve, reject) => {
      const timer = setTimeout(() => reject(new Error('dispatch timeout')), timeoutMs)
      client.on('notification', async (msg) => {
        if (!pending.has(msg.payload!)) return
        await drain()
        if (pending.size === 0) { clearTimeout(timer); resolve(results) }
      })
    })
  } finally {
    client.release()
  }
}

The ON CONFLICT … DO UPDATE … RETURNING id dance is the idempotent-insert trick — same idem_key returns the original row's id, so a dispatcher that crashes between two INSERTs and resumes won't enqueue twice. awaitAll does a drain-then-listen so completions that landed before the LISTEN registered still get picked up; without the initial drain you'd have a race where fast workers finish before the parent subscribes.

Worker: claim, run, write back

// workers/agent-worker.ts — run one per machine, or one per CPU core
import { Pool } from 'pg'
import { runSubagent } from './runner'   // your wrapper around the Claude API or CLI
const pool = new Pool({ connectionString: process.env.REDDB_URL })
const workerId = `${process.env.HOSTNAME}-${process.pid}`

async function claim() {
  const { rows } = await pool.query(`
    UPDATE agent_task
       SET state = 'running',
           worker_id = $1,
           started_at = now(),
           visible_at = now() + interval '5 minutes',
           attempts = attempts + 1
     WHERE id = (
       SELECT id FROM agent_task
        WHERE state = 'pending' AND visible_at <= now()
        ORDER BY created_at
        FOR UPDATE SKIP LOCKED
        LIMIT 1
     )
     RETURNING id, kind, payload, attempts, max_attempts
  `, [workerId])
  return rows[0]
}

async function finish(id: string, ok: boolean, body: unknown) {
  await pool.query(
    `UPDATE agent_task
        SET state = $2, ${ok ? 'result' : 'error'} = $3, finished_at = now()
      WHERE id = $1`,
    [id, ok ? 'done' : 'failed', ok ? body : String(body)],
  )
}

async function retry(id: string, attempts: number, max: number, err: unknown) {
  if (attempts >= max) return finish(id, false, err)
  const backoff = Math.min(60, 2 ** attempts) // seconds
  await pool.query(
    `UPDATE agent_task
        SET state = 'pending', visible_at = now() + ($2 || ' seconds')::interval, error = $3
      WHERE id = $1`,
    [id, backoff, String(err)],
  )
}

while (true) {
  const task = await claim()
  if (!task) { await new Promise(r => setTimeout(r, 1000)); continue }
  try {
    const out = await runSubagent(task.kind, task.payload)
    await finish(task.id, true, out)
  } catch (err) {
    await retry(task.id, task.attempts, task.max_attempts, err)
  }
}

FOR UPDATE SKIP LOCKED is the load-bearing line. Drop it and N workers serialize on the same row lock; keep it and they fan out cleanly across the queue. The visibility-timeout update inside the same statement is the lease — if this worker crashes mid-task, the row reverts to claimable after five minutes because a watchdog (next section) flips state back to pending. The exponential backoff on retry tops out at one minute so failed tasks don't park in the queue for hours.

Watchdog: reclaim leases, alert on poison pills

-- run every minute from cron or pg_cron
UPDATE agent_task
   SET state = 'pending', visible_at = now() + interval '5 seconds'
 WHERE state = 'running'
   AND started_at < now() - interval '5 minutes';

-- alert: tasks that hit max_attempts
SELECT id, kind, error, attempts
  FROM agent_task
 WHERE state = 'failed'
   AND finished_at > now() - interval '1 hour';

First query is the lease reclaimer. Second is the canary — wire it to the observability dashboard from D9 and a Slack webhook so poison pills surface within an hour instead of clogging the queue silently.

Calling it from a Claude Code session

// .claude/skills/parallel-review/run.ts
import { dispatch, awaitAll } from '../../lib/dispatch'

const parentId = process.env.CLAUDE_SESSION_ID!
const files: string[] = JSON.parse(process.env.CHANGED_FILES!)

const ids = await Promise.all(files.map(f =>
  dispatch('review', { file: f, ruleset: 'house-style' }, {
    parentId,
    idemKey: `${parentId}:${f}`,
  })
))

const results = await awaitAll(parentId, ids, 15 * 60_000)
for (const [, r] of results) {
  console.log(r.state === 'done' ? r.result : `FAIL: ${r.error}`)
}

From the model's perspective this looks like a synchronous fan-out, but the rows live in the DB. Kill the session, restart it tomorrow, re-run with the same parentId and idem_keys, and awaitAll returns immediately for tasks already finished and waits only on the stragglers. That's the durable part.

When this pattern earns its keep

Three signals that you've outgrown raw Task:

Single fan-out > 30 sub-agents. In-process parallelism starts to choke on rate limits and the parent's context fills with sub-agent output it has to scroll past.
Work outlasts a session. Anything where "let it run overnight" is a normal answer — evals, batch refactors, doc regeneration.
More than one machine has spare cycles. A laptop and a workstation can both pull from the same queue; the queue is the only piece of shared state.

If none of those hold, Task is still the right primitive — don't drag in a database to dispatch three sibling agents that finish in 90 seconds.

What's next

D13 (eval datasets) builds on this queue: every dispatched eval becomes a row, every replay against a new model version is another fan-out against the same set of payloads. D14 (cross-session task queues) generalises the worker loop into the AFK-agent runtime: same table, same claim query, plus checkpoints and resume tokens so a sub-agent can be paused mid-thought and revived on a different host.

Context window economics — when memory beats reloading docs

2026-06-01

The hidden bill on every CLI agent session

Open Claude Code in a medium repo. Ask a question. Watch it Read the same five files it read yesterday. Ask a follow-up. Watch it Grep the same constants. The model is smart, the tool calls are cheap individually, and the per-turn waste hides inside a green checkmark — so nobody adds it up. This post does the addition, then shows the fix.

The fix is a semantic-memory layer that lives in the same RedDB table the rest of Pillar D uses: read once, embed, query top-k chunks per turn instead of re-reading the whole tree. The interesting part isn't the code — it's where the breakeven sits. For most real projects it's around turn three.

The naive baseline: re-read everything

Call this the "stateless agent" baseline. Per turn the agent reloads:

CLAUDE.md and any @import-chained memory files — call it M tokens
a handful of source files for context — call it S tokens
recent diffs, git log, command output — D tokens

Per turn, input tokens: C = M + S + D. Across N turns: N · C. Cost: N · C · p_in, where p_in is the input-token price.

For Claude Opus 4.7 (Jan 2026 list) that's roughly $15 / 1M input, $75 / 1M output. Prompt cache hits land near $1.50 / 1M. Cache is the reason this isn't worse — but cache misses on every new turn that touches a slightly different file set, and the cache TTL is five minutes. After lunch you pay full price again.

A worked number for a 50k-line monorepo, ten-turn session, ~30k input tokens per turn (10k from rules + 15k from re-read files + 5k from logs):

input tokens   = 10 turns × 30_000 = 300_000
cost (cold)    = 300_000 / 1_000_000 × $15  = $4.50
cost (warm)    = 300_000 / 1_000_000 × $1.50 = $0.45

Forty-five cents per warm session sounds fine. Multiply by 12 engineers × 8 sessions/day × 21 working days and you're at $907/month, and that's the cached number. The cold-cache reality across machine restarts and overnight idle is closer to $2k/month per twelve-engineer team — for the privilege of re-reading files that haven't changed.

The memory-backed alternative

Swap the per-turn re-read for a per-turn semantic query.

Index once. A background indexer chunks every file in the repo (200–400 tokens per chunk), embeds it with voyage-3 (~$0.06 / 1M tokens, one-time amortised), writes the rows into the same memory table.
Query per turn. On SessionStart (or whichever hook owns context priming), embed the conversation tail, run a <=> cosine search, pull the top-k chunks. k = 6 and chunk size 300 tokens means ~1.8k tokens of injected context vs the 15k of "re-read the whole file" — an order of magnitude shave on the source-file portion of C.
Re-index on change. A PostToolUse hook on Write|Edit invalidates and re-embeds the touched files. Tiny per-edit cost; rounds to zero against any session.

The per-turn input now looks like C' = M + Q + R, where Q is the embedding query (a few hundred tokens of conversation tail) and R is the retrieved chunks. In our worked example, C' ≈ 10_000 + 800 + 1_800 = 12_600, down from 30k. The model still does the work — it just doesn't re-read what it already knew.

The breakeven, plotted as algebra

Per session, the memory-backed path pays:

cost_memory = N · C' · p_in                # per-turn query + retrieval
            + E_query · N · p_embed        # embedding the query each turn
            + E_index · F · p_embed / R_amort   # amortised indexing

E_index is total tokens indexed, F is the change-frequency multiplier (how often files re-embed), R_amort is how many sessions the index serves before a full rebuild. For most repos R_amort ≫ 100, and p_embed is ~250× cheaper than p_in, so the indexing column vanishes.

The naive path pays:

cost_naive = N · C · p_in

Breakeven is the turn count where the constant-cost ramp-up of the memory path is paid off by the smaller per-turn slope:

N_breakeven  =  E_index · p_embed / R_amort
                ────────────────────────────────
                (C - C') · p_in  -  E_query · p_embed

Plug the worked numbers (C - C' = 17_400 input tokens saved per turn, E_query = 800, p_in = $15/Mtok, p_embed = $0.06/Mtok, E_index = 5_000_000, R_amort = 200):

numerator    = 5_000_000 × 0.06 / 200 / 1_000_000  ≈ $0.0015
denominator  = (17_400 × 15 - 800 × 0.06) / 1_000_000 ≈ $0.000261
N_breakeven  ≈ 5.7 turns

Below six turns per session, full re-read wins. Above six, memory wins, and the gap widens linearly with N. Any AFK / Ralph-style loop blows past breakeven in the first hour.

The actual code

The schema is the one from the memory-backend post, no changes:

-- already exists from D1
CREATE TABLE memory (
  id         text PRIMARY KEY,
  scope      text NOT NULL,           -- 'repo:<slug>:file:<path>'
  body       text NOT NULL,
  embedding  vector(1024),
  meta       jsonb NOT NULL DEFAULT '{}'::jsonb,
  created_at timestamptz NOT NULL DEFAULT now()
);

CREATE INDEX memory_embedding_hnsw
  ON memory USING hnsw (embedding vector_cosine_ops);

A one-shot indexer (Node, ~40 lines, runs from the repo root):

// scripts/index-repo.ts
import { readFile } from 'node:fs/promises'
import { glob } from 'glob'
import { ulid } from 'ulid'
import postgres from 'postgres'
import { VoyageAIClient } from 'voyageai'

const sql = postgres(process.env.DATABASE_URL!)
const voyage = new VoyageAIClient({ apiKey: process.env.VOYAGE_API_KEY! })
const REPO = process.env.REPO_SLUG ?? 'rdb-lair'
const CHUNK = 300                          // ~tokens, sliced by characters at 4x

const files = await glob('**/*.{ts,tsx,md,sql}', { ignore: ['**/node_modules/**', '**/dist/**'] })

for (const path of files) {
  const text = await readFile(path, 'utf8')
  const chunks: string[] = []
  for (let i = 0; i < text.length; i += CHUNK * 4) chunks.push(text.slice(i, i + CHUNK * 4))
  if (chunks.length === 0) continue

  const embeddings = await voyage.embed({ input: chunks, model: 'voyage-3' })
  const rows = chunks.map((body, i) => ({
    id: ulid(),
    scope: `repo:${REPO}:file:${path}`,
    body,
    embedding: JSON.stringify(embeddings.data[i].embedding),
    meta: { path, chunk: i },
  }))

  await sql`INSERT INTO memory ${sql(rows, 'id', 'scope', 'body', 'embedding', 'meta')}`
  console.log(`indexed ${path} (${chunks.length} chunks)`)
}

await sql.end()

The SessionStart hook that replaces the per-turn re-read:

#!/usr/bin/env bash
# ~/.claude/hooks/session-start-context.sh — emits top-k chunks for the current task
set -euo pipefail

TAIL="$(jq -r '.conversation_tail // ""' <<<"$CLAUDE_HOOK_INPUT")"
[ -z "$TAIL" ] && exit 0

QVEC=$(curl -s https://api.voyageai.com/v1/embeddings \
  -H "Authorization: Bearer $VOYAGE_API_KEY" \
  -H 'content-type: application/json' \
  -d "$(jq -n --arg q "$TAIL" '{input:[$q], model:"voyage-3"}')" \
  | jq -c '.data[0].embedding')

psql "$DATABASE_URL" -At -F $'\t' <<SQL | jq -Rs '{context: .}'
SELECT meta->>'path' AS path, body
FROM memory
WHERE scope LIKE 'repo:${REPO_SLUG}:file:%'
ORDER BY embedding <=> '${QVEC}'::vector
LIMIT 6;
SQL

Wire it up in .claude/settings.json:

{
  "hooks": {
    "SessionStart": [
      { "command": "~/.claude/hooks/session-start-context.sh", "priority": 20 }
    ],
    "PostToolUse": [
      { "matcher": "Write|Edit", "command": "~/.claude/hooks/reindex-touched-file.sh" }
    ]
  }
}

The reindex-touched-file.sh hook re-embeds just the file that changed — same shape as the indexer above, scoped to one path. Skipped here for length; it's ~20 lines.

What to measure once it's live

Don't trust the model. Measure. The agent observability post drops a PostToolUse hook that captures every tool call with token counts and cost. Compare two weeks before and after:

SELECT
  date_trunc('day', started_at) AS day,
  sum(input_tokens)             AS input_tokens,
  sum(cost_usd)                 AS cost_usd
FROM tool_call
WHERE session_id IN (SELECT id FROM session WHERE repo = 'rdb-lair')
GROUP BY 1 ORDER BY 1;

Two failure modes to watch:

Retrieval too narrow. k = 6 chunks misses cross-cutting context. Symptom: the model asks Read for files anyway. Bump k to 12, or add a second query embedded from the user's last message instead of the rolling tail.
Stale index. PostToolUse reindex misses files changed outside Claude (git pull, another editor). Fix: nightly cron that rescans git diff HEAD~24h and re-embeds.

When this is the wrong call

Three cases where the naive re-read still wins:

Tiny repos. Under ~2k lines, C - C' is small enough that N_breakeven jumps past 20 turns. Skip the indexer; let the prompt cache handle it.
Single-shot scripts. A one-question session never amortises the per-query embedding cost. Memory layer helps zero.
High-churn refactors. When 40% of files change per session, reindex cost climbs and R_amort collapses. In that mode, fall back to plain Read for the touched tree and only query memory for the rest.

For everything else — long-running AFK loops, multi-day feature work, anything that crosses a context-compression boundary — the breakeven is six turns. Pay it once.

Where this fits in the pillar

Storage layer: Claude Code's memory backend (the table this post indexes into)
Hook scaffolding: Hooks that mutate persistent state (the PostToolUse transactional pattern that keeps re-index idempotent)
Cost telemetry: Agent observability: traces, tokens, and costs (the queries that prove the savings are real)
Forward link: sub-agent dispatch and AFK queues use the same memory table for cross-session continuity — drafts coming in the next Ralph iterations.

Multi-agent shared memory — five agents, one knowledge graph

2026-05-30

Why parallel agents need a shared scratchpad

Spawning five Claude Code sub-agents in parallel is easy. Letting them cooperate is not. Each sub-agent has its own context, its own tool budget, and no idea what its siblings just decided. Without a coordination layer you get three failure modes within the first hour:

Duplicate work. Two agents independently re-derive that the auth middleware uses JWT.
Contradictory facts. Agent A writes "rate limit is 100 rpm". Agent B, five minutes later, writes "rate limit is 60 rpm". The orchestrator merges both and the parent session uses whichever sorts last.
Lost decisions. Agent C decides to switch the queue from SQS to Redis. Nobody else hears about it until the integration step blows up.

A shared knowledge graph in RedDB — with snapshot reads and first-committer-wins writes — fixes all three. Reuses the memory table from the memory-backend post almost verbatim, plus one extra column for the agent that wrote the row and one extra table for the coordination doc.

The shape of the fix

Three primitives:

Snapshot reads. Each sub-agent reads under a fixed transaction snapshot. Whatever was committed when the agent started is what the agent sees, even if siblings keep writing.
First-committer-wins writes. Writes to the same (scope, key) are rejected for any agent that did not see the latest version. The losing agent gets the conflict back as a structured error and decides what to do — retry, escalate, or merge.
A coordination doc. A single row in consensus per topic, pointing at the latest agreed value. The orchestrator subscribes to changes and surfaces contradictions.

Schema

-- Reuse the memory table from the D1 post, add the writer.
ALTER TABLE memory ADD COLUMN written_by TEXT NOT NULL DEFAULT 'human';
ALTER TABLE memory ADD COLUMN scope_key  TEXT;                -- e.g. "auth.rate_limit"
ALTER TABLE memory ADD COLUMN version    BIGINT NOT NULL DEFAULT 1;

CREATE UNIQUE INDEX memory_scope_key_version
  ON memory (scope_key, version)
  WHERE scope_key IS NOT NULL;

-- One row per topic. Points at the memory row currently considered "true".
CREATE TABLE consensus (
  scope_key   TEXT PRIMARY KEY,
  memory_id   TEXT NOT NULL REFERENCES memory(id),
  agreed_at   TIMESTAMPTZ NOT NULL DEFAULT now(),
  agreed_by   TEXT NOT NULL,                -- orchestrator id
  version     BIGINT NOT NULL
);

-- Audit trail of every conflict so the orchestrator can replay them.
CREATE TABLE conflict (
  id          TEXT PRIMARY KEY,
  scope_key   TEXT NOT NULL,
  loser_id    TEXT NOT NULL REFERENCES memory(id),
  winner_id   TEXT NOT NULL REFERENCES memory(id),
  detected_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

The version column is the optimistic-concurrency token. The unique index over (scope_key, version) is what makes "first committer wins" enforceable in one round-trip: the second writer's INSERT fails with a unique-violation, no advisory locks, no SELECT ... FOR UPDATE.

The write path

Each sub-agent runs the same compare-and-set:

WITH latest AS (
  SELECT version FROM memory
   WHERE scope_key = $1
   ORDER BY version DESC
   LIMIT 1
)
INSERT INTO memory (id, kind, scope, scope_key, body, embedding, written_by, version)
SELECT $2, 'fact', $3, $1, $4, $5, $6, COALESCE((SELECT version FROM latest), 0) + 1
RETURNING id, version;

If two agents fire this query concurrently with the same scope_key, both compute version = N+1 and both try to insert. The unique index lets exactly one win. The other gets 23505 unique_violation back, which the agent SDK should map to a typed ConflictError carrying the winning row's id.

Code (Node, intentionally boring):

async function writeFact(scopeKey: string, body: string, agentId: string) {
  for (let attempt = 0; attempt < 3; attempt++) {
    try {
      const { rows } = await pg.query(WRITE_FACT_SQL, [
        scopeKey,
        ulid(),
        'agent:' + agentId,
        body,
        await embed(body),
        agentId,
      ])
      return { ok: true, id: rows[0].id, version: rows[0].version }
    } catch (err) {
      if (err.code !== '23505') throw err
      // Lost the race. Read the current truth and decide.
      const current = await readConsensus(scopeKey)
      if (current.body === body) return { ok: true, id: current.id, version: current.version }
      return { ok: false, conflict: current, attempted: body }
    }
  }
  throw new Error('writeFact: exhausted retries')
}

The ok: false branch is the interesting one. The losing agent now has both the consensus value and its own attempted value in hand, and has to choose:

Identical body → silently accept, no-op. (Same conclusion reached twice — common with parallel research agents.)
Different body → record a conflict and either escalate to the orchestrator or attempt a merge.

The read path

Each sub-agent reads under a snapshot transaction so that mid-task writes by siblings do not change what the agent already reasoned about:

async function readScope(scope: string) {
  const client = await pg.connect()
  try {
    await client.query('BEGIN ISOLATION LEVEL REPEATABLE READ')
    const memories = await client.query(
      `SELECT m.id, m.scope_key, m.body, m.version, m.written_by
         FROM memory m
         JOIN consensus c ON c.memory_id = m.id
        WHERE m.scope = $1`,
      [scope],
    )
    await client.query('COMMIT')
    return memories.rows
  } finally {
    client.release()
  }
}

REPEATABLE READ means subsequent reads in the same transaction see the same snapshot. For a long-running agent task that mixes reads and tool calls, opening one snapshot at the start and closing it at the end is the cleanest model — every fact the agent acts on is internally consistent, even if the world moved on.

Recording conflicts and alerting the orchestrator

The losing-write path inserts a conflict row. The orchestrator subscribes to LISTEN conflict_inserted (Postgres NOTIFY triggered from a trivial AFTER INSERT trigger) and gets a structured event the moment two agents disagree:

CREATE OR REPLACE FUNCTION notify_conflict() RETURNS trigger AS $$
BEGIN
  PERFORM pg_notify('conflict_inserted', json_build_object(
    'scope_key', NEW.scope_key,
    'loser_id',  NEW.loser_id,
    'winner_id', NEW.winner_id
  )::text);
  RETURN NEW;
END $$ LANGUAGE plpgsql;

CREATE TRIGGER conflict_inserted_notify
AFTER INSERT ON conflict
FOR EACH ROW EXECUTE FUNCTION notify_conflict();

The orchestrator-side listener is ten lines:

const client = await pg.connect()
await client.query('LISTEN conflict_inserted')
client.on('notification', (msg) => {
  const { scope_key, loser_id, winner_id } = JSON.parse(msg.payload!)
  orchestrator.alert({
    kind: 'fact_conflict',
    topic: scope_key,
    candidates: [loser_id, winner_id],
  })
})

In Claude Code the orchestrator is usually the parent session that spawned the sub-agents via the Task tool. Surface the alert as a SessionStart-style banner in the next turn so the human (or the parent agent) decides which fact to promote.

Promoting consensus

When the parent picks a winner — manually or by rule (most-recent, highest-confidence, majority-vote) — promotion is one row update:

INSERT INTO consensus (scope_key, memory_id, agreed_by, version)
VALUES ($1, $2, $3, $4)
ON CONFLICT (scope_key) DO UPDATE
   SET memory_id = EXCLUDED.memory_id,
       agreed_by = EXCLUDED.agreed_by,
       version   = EXCLUDED.version,
       agreed_at = now()
 WHERE consensus.version < EXCLUDED.version;

The WHERE consensus.version < EXCLUDED.version guard means an out-of-order promotion (rare, but possible if the orchestrator itself parallelises) never rolls consensus backwards.

Worked example

Three sub-agents auditing a service. The orchestrator spawns them in parallel:

agent-a: read auth/* → fact "rate_limit = 100 rpm"  → INSERT version 1 ✓
agent-b: read docs/*  → fact "rate_limit = 60 rpm"   → INSERT version 1 ✗ (unique violation)
                                                      → reads consensus (100 rpm)
                                                      → bodies differ, INSERT conflict
agent-c: read tests/* → fact "rate_limit = 100 rpm"  → bodies match, no-op

The orchestrator's listener fires once, receives {scope_key: "auth.rate_limit", loser: B, winner: A}, and the next turn of the parent session sees:

⚠️ Two sub-agents disagreed on auth.rate_limit. Candidate facts: agent-a says 100 rpm (from auth/limits.ts:42), agent-b says 60 rpm (from docs/api.md:88). Promote one?

The human (or a /promote slash command) picks one, and the consensus row updates. The other fact stays in memory for audit — nothing is discarded, only de-promoted.

What this gives you

Isolation per agent via REPEATABLE READ snapshots — no torn reads, no flapping facts mid-task.
One-shot conflict detection via the (scope_key, version) unique index — no advisory locks, no leader election.
Append-only history — every losing write is still in the memory table, queryable for "when did we change our mind about X?".
Push-based orchestrator alerts via LISTEN/NOTIFY — no polling, conflicts surface within milliseconds.

The whole thing is roughly 150 lines of SQL + TypeScript and rides on the same RedDB instance already wired up for agent observability and the /remember command. When the next post in this pillar adds sub-agent dispatch via a database queue, the queue rows will simply point at scope keys here.

Five agents, one knowledge graph, no merge hell.

Agent observability — traces, tokens, and costs in RedDB

2026-05-28

The questions you can't answer today

Run Claude Code daily for a week and you accumulate a folder of transcripts, a credit-card bill from Anthropic, and no honest answer to questions a manager will ask within a month:

How much did the team spend on agent runs last sprint?
Which tools account for the bulk of token usage — Read, Bash, sub-agents?
What is the p99 latency of Bash vs Read? Is one of them dragging turn times?
Which sessions blew past a sensible cost budget, and what were they doing?
After we shipped the new memory hook, did average turn cost go up or down?

Each of these is a GROUP BY away once the data is in a table. The work is in getting it there, accurately, without slowing the agent down. A PostToolUse hook plus three tables is enough.

This post assumes you have the same RedDB-as-memory-backend setup from the first post in this pillar and the transactional hook recipe from the audit-log post. It reuses the idempotency pattern verbatim — observability is a special case of structured audit.

The schema

Three tables. One for the tool call itself, one for token/cost facts emitted by the model API, one for session-level rollups so the dashboard does not re-aggregate twelve million rows on every page load.

CREATE TABLE tool_call (
  id              TEXT PRIMARY KEY,                  -- ulid
  session_id      TEXT NOT NULL,
  turn_index      INTEGER NOT NULL,                  -- monotonic within session
  tool_name       TEXT NOT NULL,                     -- Read, Bash, Skill, Task, …
  tool_input_size INTEGER NOT NULL,                  -- bytes of tool_input json
  is_error        BOOLEAN NOT NULL DEFAULT false,
  started_at      TIMESTAMPTZ NOT NULL,
  ended_at        TIMESTAMPTZ NOT NULL,
  duration_ms     INTEGER GENERATED ALWAYS AS
                    (EXTRACT(EPOCH FROM (ended_at - started_at)) * 1000)::int STORED
);

CREATE TABLE token_usage (
  id              TEXT PRIMARY KEY,                  -- ulid
  session_id      TEXT NOT NULL,
  turn_index      INTEGER NOT NULL,
  model           TEXT NOT NULL,                     -- claude-opus-4-7, …
  input_tokens    INTEGER NOT NULL,
  cached_tokens   INTEGER NOT NULL DEFAULT 0,        -- prompt-cache hits
  output_tokens   INTEGER NOT NULL,
  cost_usd        NUMERIC(10,6) NOT NULL,            -- computed from the model price list
  recorded_at     TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE TABLE session_rollup (
  session_id      TEXT PRIMARY KEY,
  started_at      TIMESTAMPTZ NOT NULL,
  last_seen_at    TIMESTAMPTZ NOT NULL,
  total_turns     INTEGER NOT NULL DEFAULT 0,
  total_tools     INTEGER NOT NULL DEFAULT 0,
  total_cost_usd  NUMERIC(10,6) NOT NULL DEFAULT 0,
  project_path    TEXT,
  user_email      TEXT
);

CREATE INDEX tool_call_session_idx ON tool_call (session_id, turn_index);
CREATE INDEX tool_call_name_idx    ON tool_call (tool_name, started_at DESC);
CREATE INDEX token_usage_session_idx ON token_usage (session_id, turn_index);

tool_call is one row per PostToolUse event. token_usage is one row per assistant turn (the model API returns the counts in its response, which the harness exposes via Stop and PostToolUse payloads on the Anthropic adapter). session_rollup is maintained by trigger so the dashboard reads constant-time per session.

The PostToolUse hook

The hook is the audit-log hook from the previous post with three extra columns and a token-usage insert in the same transaction. The idempotency key is the same sha256(session_id + tool_use_id).

#!/usr/bin/env bash
# ~/.claude/hooks/observe.sh
set -euo pipefail

payload="$(cat)"

session_id="$(jq  -r '.session_id'                <<<"$payload")"
turn_index="$(jq  -r '.turn_index // 0'           <<<"$payload")"
tool_use_id="$(jq -r '.tool_use_id // .id // ""'  <<<"$payload")"
tool_name="$(jq   -r '.tool_name // ""'           <<<"$payload")"
tool_input="$(jq    '.tool_input // {}'           <<<"$payload")"
is_error="$(jq    -r '.is_error // false'         <<<"$payload")"
started_at="$(jq  -r '.started_at'                <<<"$payload")"
ended_at="$(jq    -r '.ended_at'                  <<<"$payload")"

# Token usage is only present on Stop and on the turn-ending PostToolUse;
# pull it defensively.
model="$(jq         -r '.usage.model // ""'         <<<"$payload")"
in_tokens="$(jq     -r '.usage.input_tokens   // 0' <<<"$payload")"
cached_tokens="$(jq -r '.usage.cached_tokens  // 0' <<<"$payload")"
out_tokens="$(jq    -r '.usage.output_tokens  // 0' <<<"$payload")"

tool_id="$(ulid)"
idem_key="$(printf '%s:%s' "$session_id" "$tool_use_id" | sha256sum | awk '{print $1}')"
input_size="$(printf '%s' "$tool_input" | wc -c)"

# Price table lives in the hook so a model-list change is one diff.
cost_usd="$(python3 - <<PY
prices = {
  "claude-opus-4-7":   (15.00 / 1e6, 75.00 / 1e6, 1.50 / 1e6),  # in, out, cached-read
  "claude-sonnet-4-6": ( 3.00 / 1e6, 15.00 / 1e6, 0.30 / 1e6),
  "claude-haiku-4-5":  ( 0.80 / 1e6,  4.00 / 1e6, 0.08 / 1e6),
}
m = "$model"
if m not in prices or ($in_tokens + $out_tokens) == 0:
  print("0")
else:
  pi, po, pc = prices[m]
  print(f"{$in_tokens * pi + $out_tokens * po + $cached_tokens * pc:.6f}")
PY
)"

curl -sS --fail --max-time 1 \
  -H "Authorization: Bearer $REDDB_TOKEN" \
  -H "Content-Type: application/json" \
  "$REDDB_URL/tx" \
  -d "$(jq -nc \
        --arg tid       "$tool_id" \
        --arg sid       "$session_id" \
        --argjson turn  "$turn_index" \
        --arg tname     "$tool_name" \
        --argjson isz   "$input_size" \
        --argjson err   "$is_error" \
        --arg sat       "$started_at" \
        --arg eat       "$ended_at" \
        --arg model     "$model" \
        --argjson it    "$in_tokens" \
        --argjson ct    "$cached_tokens" \
        --argjson ot    "$out_tokens" \
        --arg cost      "$cost_usd" \
        --arg ikey      "$idem_key" \
        '{
          statements: [
            { sql: "INSERT INTO tool_call (id, session_id, turn_index, tool_name, tool_input_size, is_error, started_at, ended_at) VALUES ($1,$2,$3,$4,$5,$6,$7,$8) ON CONFLICT (id) DO NOTHING",
              args: [$tid, $sid, $turn, $tname, $isz, $err, $sat, $eat] },
            { sql: "INSERT INTO hook_idempotency (key, event_id) VALUES ($1,$2) ON CONFLICT (key) DO NOTHING",
              args: [$ikey, $tid] },
            { sql: "INSERT INTO token_usage (id, session_id, turn_index, model, input_tokens, cached_tokens, output_tokens, cost_usd) SELECT ulid(), $1, $2, $3, $4, $5, $6, $7 WHERE $3 <> ''",
              args: [$sid, $turn, $model, $it, $ct, $ot, $cost] }
          ]
        }')" >/dev/null || true

echo '{"continue":true}'

Two notes:

--max-time 1 + || true — the agent never blocks on telemetry. If RedDB is unreachable for a second, that one event is lost. A Stop hook can backfill from the harness's local JSONL transcript if you need full coverage.
The third statement is gated on $model <> ''. Most PostToolUse events do not carry usage data — only the turn-closing one does. The conditional INSERT … SELECT … WHERE skips the row at SQL level rather than branching in bash.

Wire it up in ~/.claude/settings.json:

{
  "hooks": {
    "PostToolUse": [
      { "matcher": ".*", "command": "~/.claude/hooks/observe.sh" }
    ],
    "Stop": [
      { "matcher": ".*", "command": "~/.claude/hooks/observe.sh" }
    ]
  }
}

The rollup trigger

Reading raw tool_call and token_usage for "show me last sprint's cost by user" is fine at 100k rows and miserable at 10M. A trigger keeps session_rollup warm:

CREATE OR REPLACE FUNCTION bump_rollup() RETURNS trigger AS $$
BEGIN
  INSERT INTO session_rollup (session_id, started_at, last_seen_at, total_turns, total_tools, total_cost_usd)
  VALUES (NEW.session_id, NEW.started_at, NEW.ended_at, 0, 1, 0)
  ON CONFLICT (session_id) DO UPDATE
     SET last_seen_at = GREATEST(session_rollup.last_seen_at, EXCLUDED.last_seen_at),
         total_tools  = session_rollup.total_tools + 1;
  RETURN NEW;
END $$ LANGUAGE plpgsql;

CREATE TRIGGER tool_call_rollup AFTER INSERT ON tool_call
FOR EACH ROW EXECUTE FUNCTION bump_rollup();

CREATE OR REPLACE FUNCTION bump_cost() RETURNS trigger AS $$
BEGIN
  UPDATE session_rollup
     SET total_cost_usd = total_cost_usd + NEW.cost_usd,
         total_turns    = GREATEST(total_turns, NEW.turn_index + 1)
   WHERE session_id = NEW.session_id;
  RETURN NEW;
END $$ LANGUAGE plpgsql;

CREATE TRIGGER token_usage_rollup AFTER INSERT ON token_usage
FOR EACH ROW EXECUTE FUNCTION bump_cost();

session_rollup is now the queryable summary. The raw tables stay for drill-down.

The five queries that earn the hook

-- 1. Sprint spend by user.
SELECT user_email,
       SUM(total_cost_usd)::numeric(10,2) AS cost_usd,
       COUNT(*)                            AS sessions
  FROM session_rollup
 WHERE started_at >= date_trunc('day', now()) - interval '14 days'
 GROUP BY user_email
 ORDER BY cost_usd DESC;

-- 2. Top tools by total token-equivalent cost (rough — tools don't pay tokens
--    directly, but tool_input bytes correlate with the prompt cost they cause).
SELECT tool_name,
       COUNT(*)                                          AS calls,
       SUM(tool_input_size)                              AS bytes_into_prompt,
       SUM(tool_input_size) * 0.000003                   AS approx_cost_usd
  FROM tool_call
 WHERE started_at >= now() - interval '7 days'
 GROUP BY tool_name
 ORDER BY approx_cost_usd DESC
 LIMIT 10;

-- 3. p99 latency by tool, last 24h.
SELECT tool_name,
       percentile_disc(0.50) WITHIN GROUP (ORDER BY duration_ms) AS p50,
       percentile_disc(0.99) WITHIN GROUP (ORDER BY duration_ms) AS p99,
       COUNT(*)                                                   AS n
  FROM tool_call
 WHERE started_at >= now() - interval '24 hours'
 GROUP BY tool_name
 ORDER BY p99 DESC;

-- 4. Sessions over a $5 budget, this week.
SELECT session_id, user_email, total_cost_usd, total_turns, total_tools, project_path
  FROM session_rollup
 WHERE started_at >= date_trunc('week', now())
   AND total_cost_usd > 5.00
 ORDER BY total_cost_usd DESC
 LIMIT 25;

-- 5. Before/after comparison for a hook change deployed at 2026-05-20.
SELECT date_trunc('day', started_at) AS day,
       AVG(total_cost_usd)::numeric(10,4) AS avg_session_cost,
       COUNT(*)                            AS sessions
  FROM session_rollup
 WHERE started_at >= '2026-05-13'
 GROUP BY day
 ORDER BY day;

Each one is the smallest possible question to ask of the schema. EXPLAIN ANALYZE on a year of single-user data on commodity hardware runs all five in under 50ms thanks to the rollup table and the two indexes on tool_call.

A dashboard sketch

The five queries above are also the five panels of a sensible "agent observability" board. Sketched in Grafana terms but the pattern translates to anything that speaks SQL:

┌──────────────────────────────────────────────────────────────────┐
│  Sprint spend by user        │  Top tools by approx cost          │
│  (bar — query 1)             │  (bar — query 2)                   │
├──────────────────────────────┼────────────────────────────────────┤
│  p50 / p99 by tool, 24h      │  Sessions over $5 this week        │
│  (heatmap — query 3)         │  (table — query 4)                 │
├──────────────────────────────┴────────────────────────────────────┤
│  Daily average session cost (line — query 5, annotated with        │
│  vertical markers at each hook deploy)                             │
└────────────────────────────────────────────────────────────────────┘

The vertical annotations on the last panel are the highest-leverage piece. Every time you deploy a new hook, skill, or system prompt, drop a marker. The line either bends down (the change paid off) or it does not. Most teams ship agent changes blind because the feedback loop is "vibes". This panel is the feedback loop.

What this buys you

Question	Before	After
Sprint spend by user	Ask Anthropic billing CSV, grep for emails	`SELECT … FROM session_rollup`
Which tools dominate latency	Read transcripts, eyeball	One `percentile_disc` query
Did the new skill make turns cheaper	Hope so	Annotated time-series, signed answer
Why did session X cost $40	Open the transcript, scroll	`SELECT * FROM tool_call WHERE session_id = …`

The hook is ~50 lines. The schema is three tables and two triggers. The dashboard is one Grafana JSON away. The payoff is that "how is the agent doing" stops being a vibes question and starts being a query.

What is next in this pillar

The same tool_call table powers a context-window economics post: knowing the bytes-into-prompt per tool lets you compute the dollar cost of re-reading a file vs. caching it in RedDB.
A sub-agent dispatch queue post that reuses tool_call.tool_name = 'Task' rows to track parent/child runtime and cost across processes.
An MCP-server post that exposes observability.* to other CLI agents — Cursor and Codex emit the same tool-call shape, so the schema generalises.

The Atom feed carries them when they land.

An MCP server for RedDB — one database, every agent CLI

2026-05-26

Why MCP is the right seam

Earlier posts in this pillar wired RedDB into Claude Code directly: a SessionStart hook for retrieval, a skill for writes, slash commands for human-driven edits. Each of those is a Claude Code-shaped integration — a hook config, a SKILL.md, a markdown file in ~/.claude/commands/. None of it survives a port to Codex or Cursor.

The Model Context Protocol moves the seam one level down. An MCP server exposes a typed set of tools over stdio (or HTTP/SSE), and any MCP-aware host can mount it. Write the integration once, register it in every host's config, and the same memory.write tool is available from every agent CLI you run. The host's harness handles the rest — tool advertisement, schema validation, call routing.

For RedDB this maps cleanly. The database already speaks four data models (KV, document, vector, graph); MCP wants a flat list of tools with JSON Schema. We pick the four operations that pay rent in a daily agent loop:

MCP tool	RedDB operation	Used for
`memory.write`	`INSERT` into `memory` table with embedding	Capturing a fact mid-turn
`memory.search`	`SELECT … ORDER BY embedding <=> $1`	Pulling top-k context on demand
`doc.upsert`	Document store write, keyed by path	Saving a generated artifact (spec, ADR)
`vector.search`	Raw HNSW query against a named index	RAG over a corpus the agent doesn't own

Everything else (transactions, graph traversals, multi-statement SQL) stays behind the server. Tools should be small.

The manifest

MCP servers are described by a tiny JSON manifest. The host reads it once and uses the tools list to populate the model's tool registry on every turn. Schemas are JSON Schema 2020-12.

{
  "name": "reddb",
  "version": "0.1.0",
  "description": "RedDB memory and document storage for agent CLIs.",
  "tools": [
    {
      "name": "memory.write",
      "description": "Persist a short fact the agent wants to remember across sessions. Embeds the body and writes a row to the memory table.",
      "inputSchema": {
        "type": "object",
        "required": ["body"],
        "properties": {
          "body": { "type": "string", "maxLength": 2000 },
          "tags": { "type": "array", "items": { "type": "string" } },
          "project": { "type": "string" }
        }
      }
    },
    {
      "name": "memory.search",
      "description": "Semantic search over previously remembered facts. Returns top-k rows ranked by cosine similarity.",
      "inputSchema": {
        "type": "object",
        "required": ["query"],
        "properties": {
          "query": { "type": "string" },
          "k": { "type": "integer", "minimum": 1, "maximum": 20, "default": 5 },
          "project": { "type": "string" }
        }
      }
    },
    {
      "name": "doc.upsert",
      "description": "Write or replace a document at the given path. Body is plain text or markdown.",
      "inputSchema": {
        "type": "object",
        "required": ["path", "body"],
        "properties": {
          "path": { "type": "string" },
          "body": { "type": "string" }
        }
      }
    },
    {
      "name": "vector.search",
      "description": "Raw similarity search against a named HNSW index. Use for RAG over corpora not owned by the agent (codebases, docs).",
      "inputSchema": {
        "type": "object",
        "required": ["index", "query"],
        "properties": {
          "index": { "type": "string" },
          "query": { "type": "string" },
          "k": { "type": "integer", "minimum": 1, "maximum": 50, "default": 10 }
        }
      }
    }
  ]
}

A few choices worth flagging. project is optional on memory.* because the server defaults it from an environment variable — the host sets REDDB_PROJECT=rdb-lair once and every call inherits it. The model never has to remember which project it's in. body is capped at 2000 characters; longer artifacts go through doc.upsert. vector.search doesn't auto-embed — the index name dictates the embedding model, and the server handles it server-side, so the model passes a plain string.

The server

A minimal implementation in Node, using the official @modelcontextprotocol/sdk and pg for RedDB's Postgres-wire protocol. Roughly 80 lines.

import { Server } from "@modelcontextprotocol/sdk/server/index.js"
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js"
import { CallToolRequestSchema, ListToolsRequestSchema } from "@modelcontextprotocol/sdk/types.js"
import pg from "pg"
import { readFileSync } from "node:fs"

const manifest = JSON.parse(readFileSync(new URL("./manifest.json", import.meta.url), "utf8"))
const project = process.env.REDDB_PROJECT ?? "default"
const db = new pg.Pool({ connectionString: process.env.REDDB_URL })

async function embed(text: string): Promise<number[]> {
  const r = await fetch("https://api.voyageai.com/v1/embeddings", {
    method: "POST",
    headers: {
      "content-type": "application/json",
      authorization: `Bearer ${process.env.VOYAGE_API_KEY}`,
    },
    body: JSON.stringify({ model: "voyage-3", input: text }),
  })
  const json = await r.json()
  return json.data[0].embedding
}

const handlers: Record<string, (args: any) => Promise<unknown>> = {
  "memory.write": async ({ body, tags = [], project: p = project }) => {
    const v = await embed(body)
    const { rows } = await db.query(
      "INSERT INTO memory (project, body, tags, embedding) VALUES ($1,$2,$3,$4) RETURNING id, created_at",
      [p, body, tags, JSON.stringify(v)],
    )
    return rows[0]
  },
  "memory.search": async ({ query, k = 5, project: p = project }) => {
    const v = await embed(query)
    const { rows } = await db.query(
      `SELECT id, body, tags, created_at, 1 - (embedding <=> $1) AS score
       FROM memory
       WHERE project = $2 AND deleted_at IS NULL
       ORDER BY embedding <=> $1
       LIMIT $3`,
      [JSON.stringify(v), p, k],
    )
    return rows
  },
  "doc.upsert": async ({ path, body }) => {
    const { rows } = await db.query(
      `INSERT INTO doc (path, body, project) VALUES ($1,$2,$3)
       ON CONFLICT (project, path) DO UPDATE SET body = EXCLUDED.body, updated_at = now()
       RETURNING path, updated_at`,
      [path, body, project],
    )
    return rows[0]
  },
  "vector.search": async ({ index, query, k = 10 }) => {
    const { rows } = await db.query(
      "SELECT * FROM reddb.vector_search($1, $2, $3)",
      [index, query, k],
    )
    return rows
  },
}

const server = new Server({ name: manifest.name, version: manifest.version }, { capabilities: { tools: {} } })

server.setRequestHandler(ListToolsRequestSchema, async () => ({ tools: manifest.tools }))

server.setRequestHandler(CallToolRequestSchema, async (req) => {
  const fn = handlers[req.params.name]
  if (!fn) throw new Error(`unknown tool: ${req.params.name}`)
  const result = await fn(req.params.arguments ?? {})
  return { content: [{ type: "text", text: JSON.stringify(result) }] }
})

await server.connect(new StdioServerTransport())

Things to notice:

Schema lives in manifest.json, not in code. The ListTools handler ships the manifest verbatim. One file is the source of truth; hosts and humans read the same thing.
embed is one HTTP call. No batching, no retries. The model only calls these tools when it has decided a fact is worth a round-trip; latency budget is generous.
Errors propagate. If the database rejects the insert, the SDK turns the thrown Error into a tool error the model sees on the next turn. No silent failures.
vector.search defers to a SQL function. The server doesn't pick an embedding model; the function reddb.vector_search looks up the index's configured model and does the right thing. Keeps the tool surface tiny.

Run it once locally with node --env-file=.env server.js. Wrong env var names will surface as a connect-time error; nothing about an MCP server is mysterious once you've started it.

Wiring into Claude Code

Claude Code reads MCP servers from ~/.claude/mcp.json (global) or .mcp.json at the repo root (per-project). A project-scoped config keeps secrets pinned to one tree:

{
  "mcpServers": {
    "reddb": {
      "command": "node",
      "args": ["./tools/reddb-mcp/server.js"],
      "env": {
        "REDDB_URL": "postgres://reddb@localhost:5432/lair",
        "REDDB_PROJECT": "rdb-lair",
        "VOYAGE_API_KEY": "${env:VOYAGE_API_KEY}"
      }
    }
  }
}

${env:…} references inherit from the user's shell, so the file is checkable into the repo without leaking secrets. After saving, restart the harness and /mcp lists the server with its four tools. The model can now call memory.write directly — no skill, no slash command, no hook.

The same config syntax works for Codex (~/.codex/mcp.json) and Cursor (settings UI, JSON-equivalent). Gemini CLI uses a slightly different shape but the same command/args/env triplet. One server, four CLIs.

What you gave up, and what you bought

You gave up Claude Code-specific affordances. A skill can include a long SKILL.md describing when to use it; an MCP tool gets one description line. A slash command can be invoked deterministically by the user; an MCP tool is only ever called by the model. If your write path needs human-in-the-loop confirmation, the slash command from the D4 post is still the right tool.

What you bought is portability and a single audit point. Every read and every write goes through one process; logging it is a console.error away. Every CLI on the team can share one memory store without each engineer maintaining four parallel hook configs. And the next agent CLI someone in the org wants to try — the one that hasn't been built yet — gets RedDB for free, the day it ships MCP support.

That is the whole pitch: the database is the integration, the protocol is the bus, the CLIs are interchangeable.

Multi-agent shared memory — the same MCP server, but with several agents writing into it concurrently and a conflict-resolution doc keeping them honest.
Agent observability — wrap the MCP server with a tool-call audit log and build the "what did Claude Code cost me last sprint" query.

Building a "remember this" skill end-to-end

2026-05-24

What we're building

Three earlier posts in this pillar set the table:

RedDB as Claude Code's memory backend — schema, SessionStart hook, retrieval query.
Slash commands with memory — /remember, /recall, /forget as the human-driven write path.
Skills as data — skill metadata and runs as queryable rows.

A slash command needs a human to type it. A SessionStart hook needs a session boundary. Neither catches the moment in the middle of a turn where a fact deserves to be remembered — a correction, a constraint, a piece of domain language the agent just learned. That is what a skill is for: the harness picks it up by description, the agent invokes it without ceremony, the side effect lands in the database.

By the end of this post you have a skill called remember-this:

Reader installs three files (SKILL.md, capture.sh, an embedder).
Mid-turn, agent decides "this is worth keeping" and invokes the skill with a short body.
Skill embeds the body, writes it to the memory table from D1, returns a confirmation.
Next session, the SessionStart hook surfaces the row in context if it scores against the active task.

Total moving parts: one markdown file, one shell script, one HTTP call. No new infrastructure beyond the RedDB instance from D1.

Prerequisites

A RedDB instance reachable from your workstation. The examples assume http://localhost:7878 and a bearer token in $REDDB_TOKEN.
The memory table from the D1 post. Repeated here so this post stands alone:

CREATE TABLE memory (
  id          TEXT PRIMARY KEY,
  kind        TEXT NOT NULL,
  scope       TEXT NOT NULL,
  body        TEXT NOT NULL,
  embedding   VECTOR(1024) NOT NULL,
  created_at  TIMESTAMPTZ NOT NULL DEFAULT now(),
  deleted_at  TIMESTAMPTZ
);

CREATE INDEX memory_scope_idx     ON memory (scope, created_at DESC) WHERE deleted_at IS NULL;
CREATE INDEX memory_embedding_idx ON memory USING hnsw (embedding vector_cosine_ops) WHERE deleted_at IS NULL;

An embedding endpoint. The examples call voyage-3 via the Voyage HTTP API; swap in any 1024-dim model and the code is the same. Set $VOYAGE_KEY.
Claude Code installed with skills enabled (any recent release).

File 1 — `SKILL.md`

Drop at ~/.claude/skills/remember-this/SKILL.md:

---
name: remember-this
description: Persist a fact, correction, or constraint into long-term memory mid-turn. Use when the user states a preference, corrects an approach, names a non-obvious constraint, or shares a piece of domain knowledge that should outlive the current session. Do not invoke for ephemeral task state, code changes, or anything already obvious from the repo. Invoke at most once per logical fact.
---

# remember-this

When this skill matches, capture the fact you just learned.

## Steps

1. Decide what to save. One memory equals one self-contained fact. Strip surrounding chat. Rephrase in third person if the user said "I" — future-you will not know who "I" was.
2. Pick a `kind`:
   - `user` — durable facts about the user (role, expertise, tools they own)
   - `feedback` — corrections to your approach, with the *why*
   - `project` — facts about the current project that are not derivable from the code
   - `reference` — pointers to external systems (Linear projects, Grafana boards, Slack channels)
3. Pick a `scope`:
   - `global` — true regardless of project
   - the current working directory — true here only
4. Run the capture:

   ```bash
   ~/.claude/skills/remember-this/capture.sh \
     --kind=<kind> \
     --scope=<scope> \
     --body="<the fact, one sentence>"

Report back to the user in one line: Remembered (<kind>, <scope>): <body>.

Do not

Save the verbatim user message. Save the fact extracted from it.
Save anything that would be obvious from reading git log or CLAUDE.md.
Save more than one row per turn. If two facts appeared, invoke twice with two bodies.
Invoke when the user is mid-task and just venting. Memory is for things that change future behavior.


Two things the description does, that matter for trigger quality:

- It tells the harness *when not* to fire. Skills with vague descriptions fire on every turn and the user disables them within a day.
- It encodes the unit of work: one fact per invocation. The agent will otherwise dump entire chat windows into a single row and the embedding becomes useless.

## File 2 — `capture.sh`

Drop at `~/.claude/skills/remember-this/capture.sh`. `chmod +x` it.

```bash
#!/usr/bin/env bash
set -euo pipefail

KIND=user
SCOPE=$(pwd)
BODY=

for arg in "$@"; do
  case "$arg" in
    --kind=*)  KIND="${arg#--kind=}"  ;;
    --scope=*) SCOPE="${arg#--scope=}" ;;
    --body=*)  BODY="${arg#--body=}"  ;;
    *) echo "unknown arg: $arg" >&2; exit 2 ;;
  esac
done

if [[ -z "$BODY" ]]; then
  echo "--body is required" >&2
  exit 2
fi

case "$KIND" in
  user|feedback|project|reference) ;;
  *) echo "invalid --kind: $KIND" >&2; exit 2 ;;
esac

ID=$(python3 -c 'import secrets,time;print(f"{int(time.time()*1000):x}{secrets.token_hex(6)}")')

EMBEDDING=$(
  curl -sS https://api.voyageai.com/v1/embeddings \
    -H "Authorization: Bearer ${VOYAGE_KEY}" \
    -H 'Content-Type: application/json' \
    -d "$(jq -nc --arg input "$BODY" '{model:"voyage-3", input:[$input]}')" \
  | jq -c '.data[0].embedding'
)

curl -sS -X POST "${REDDB_URL:-http://localhost:7878}/sql" \
  -H "Authorization: Bearer ${REDDB_TOKEN}" \
  -H 'Content-Type: application/json' \
  -d "$(jq -nc \
        --arg id "$ID" \
        --arg kind "$KIND" \
        --arg scope "$SCOPE" \
        --arg body "$BODY" \
        --argjson emb "$EMBEDDING" \
        '{sql:"INSERT INTO memory (id,kind,scope,body,embedding) VALUES ($1,$2,$3,$4,$5)", params:[$id,$kind,$scope,$body,$emb]}')" \
  >/dev/null

echo "ok $ID"

The script is intentionally small. The contract with the model is "exit zero on success, non-zero with a message on failure", so any breakage shows up in the agent's turn as an error it can recover from (re-invoke, or apologise and continue). Three things to notice:

set -euo pipefail makes silent failure impossible. A skill that pretends to save and doesn't is the worst class of bug — the user trusts the memory exists and finds out three weeks later it never did.
ID is a millisecond-precision timestamp prefix plus six random bytes. Sortable, unique, no extra ULID dependency.
The INSERT is parameterised. The body comes from a language model; treat it like user input from the open internet.

File 3 — nothing (read-back already exists)

The SessionStart hook from D1 already reads top-K rows from memory scoped to the project. Once capture.sh writes a row, the next session sees it without any change to the read path.

If the D1 hook is not in place, the minimal version:

#!/usr/bin/env bash
# ~/.claude/hooks/session-start.sh
EMBED=$(printf '%s' "$CLAUDE_PROJECT_DIR" | embed-cli)
curl -sS "${REDDB_URL}/sql" \
  -H "Authorization: Bearer ${REDDB_TOKEN}" \
  -d "$(jq -nc --arg s "$CLAUDE_PROJECT_DIR" --argjson e "$EMBED" \
        '{sql:"SELECT body FROM memory WHERE scope IN ($1,$'global') AND deleted_at IS NULL ORDER BY embedding <=> $2 LIMIT 8", params:[$s,$e]}')" \
  | jq -r '.rows[] | "- " + .body'

That output is appended to the session's initial system message. The agent now opens every turn with the most relevant rows from prior conversations.

Testing it

Start a fresh session in any project. Steer the agent into territory worth remembering:

"I'm a Go engineer working on the React side of this repo for the first time. Frame frontend explanations in terms of backend analogues."

A correctly-tuned remember-this should fire once and produce something like:

Remembered (user, /home/you/work/this-repo):
  User is a Go engineer new to React in this repo;
  prefers frontend concepts framed as backend analogues.

Verify directly against the database:

SELECT id, kind, scope, body
FROM memory
WHERE scope = '/home/you/work/this-repo'
ORDER BY created_at DESC
LIMIT 5;

Then start a new session in the same directory. The SessionStart hook should pull the row, and a question like "how should I structure this component?" should get answered in backend-analogue terms without you re-stating the preference.

What goes wrong in week one

Three failure modes, observed across the first dozen installs:

Over-firing. The skill description is the only thing the harness sees. If it reads as "save anything interesting", the agent saves chat noise. Tighten the description until the skill fires roughly once per session of real work. Re-read the Do not block above — most of it exists because of an earlier over-firing rev.

Embedding cost surprise. Every /remember is one embedding call. At Voyage's price it is fractions of a cent, but it is non-zero and it lives on the read path of every future session. If memory grows past ~10k rows for a single scope, prune. A /forget --before=30d --kind=feedback cleanup pass once a quarter keeps the index hot.

Scope confusion. scope=global looks tempting and is almost always wrong. A preference that holds for one project rarely holds for the next. Default to the project path; promote to global only after the same fact has been re-saved across three projects.

Why this shape

The skill is twenty-odd lines of markdown plus a thirty-line shell script. It deliberately does not include:

A daemon. The skill runs synchronously on the agent's turn. Latency is one HTTP roundtrip to the embedder and one to RedDB, well under a second.
A queue. If the write fails, the agent sees the non-zero exit and tells the user. No silent retry, no dropped rows.
An ORM. The SQL is short enough to read; the parameter binding is the only safety bit that matters.

The whole pillar's argument is that a relational + vector database is a sufficient memory substrate for agent CLIs. This skill is the smallest end-to-end demonstration of that claim: one table, three files, real recall on the next session.

The next post in this pillar takes the same memory table and adds the observability layer — every tool call, token, and cost landing in RedDB via a PostToolUse hook, queryable as "which skill cost me the most last sprint" and "p99 latency by tool". The remember-this skill becomes one row among thousands in that view; the database stops being a memory store and starts being the agent's substrate.

Slash commands with memory — commands that learn between sessions

2026-05-22

Stateless by default

A Claude Code slash command is, mechanically, a markdown file in ~/.claude/commands/. When the user types /foo, the harness loads foo.md, expands any !shell lines, substitutes $ARGUMENTS, and feeds the result to the model as the next user turn. That is the whole feature.

Useful, but the surface is intentionally thin: a command is a function call, not an object with state. Two consequences:

Anything the command "knows" has to be encoded in its prompt or computed by its shell expansion every time it runs.
Two invocations of the same command, ten minutes apart, share nothing. The harness has no slot for command-local state.

For commands that do one thing — /init, /test, /review — the statelessness is fine. For commands that should accumulate context across a project's lifetime — anything in the "memory" family — it is the wrong default. A /remember that forgets between sessions is just a comment.

The previous post in this pillar introduced a memory table and a SessionStart hook that reads top-K rows into context on boot. This post finishes the picture: three slash commands — /remember, /recall, /forget — that turn that table into something the agent can write, query, and prune from inside a turn.

The schema, recapped

Same table as before. Repeated here so this post stands alone:

CREATE TABLE memory (
  id          TEXT PRIMARY KEY,           -- ulid
  kind        TEXT NOT NULL,              -- user | feedback | project | reference
  scope       TEXT NOT NULL,              -- global | <project-path> | <session-id>
  body        TEXT NOT NULL,
  embedding   VECTOR(1024) NOT NULL,
  created_at  TIMESTAMPTZ NOT NULL DEFAULT now(),
  deleted_at  TIMESTAMPTZ                  -- soft delete; /forget never DROPs
);

CREATE INDEX memory_scope_idx     ON memory (scope, created_at DESC) WHERE deleted_at IS NULL;
CREATE INDEX memory_embedding_idx ON memory USING hnsw (embedding vector_cosine_ops) WHERE deleted_at IS NULL;

Two changes from the D1 post:

deleted_at is nullable and exclusive in the indexes. /forget flips the column; it does not DELETE. Memory is auditable; deletions are recoverable.
The indexes are partial. A pruned row stops costing query time the moment it is forgotten.

`/remember` — the write

The D1 post showed a minimal version. The production-shape one is below. Drop at ~/.claude/commands/remember.md:

---
description: Persist a fact, preference, or correction into long-term memory
argument-hint: [--kind=feedback] [--scope=project|global] <body...>
---

Save the following to memory. Parse flags from `$ARGUMENTS`; default `kind=user`, `scope=project` (the current working directory).

After the write succeeds, echo back the id and kind so the user has a handle for `/forget`.

!set -euo pipefail; \
  body="$ARGUMENTS"; \
  kind="$(printf '%s' "$body" | grep -oP -- '--kind=\K\S+' || echo user)"; \
  scope="$(printf '%s' "$body" | grep -oP -- '--scope=\K\S+' || echo "$PWD")"; \
  clean="$(printf '%s' "$body" | sed -E 's/--(kind|scope)=\S+ ?//g')"; \
  curl -sS --fail --max-time 3 \
    -H "Authorization: Bearer $REDDB_TOKEN" \
    -H "Content-Type: application/json" \
    "$REDDB_URL/memory/insert" \
    -d "$(jq -nc \
          --arg kind  "$kind" \
          --arg scope "$scope" \
          --arg body  "$clean" \
          '{kind:$kind, scope:$scope, body:$body}')"

The expansion shells out, the API computes the embedding server-side, the row commits, and the JSON comes back as part of the user turn the model sees. The model then narrates the success ("Saved as mem_01H7… (kind=feedback)") because the prompt told it to.

The reason /remember is one curl call and not three is the same reason MEMORY.md lost: a database is transactional. Insert + embed + index either all commit or none of them do. There is no failure mode where the row is present but unembeddable.

`/recall` — the read

/recall <query> is the symmetric command: turn a natural-language question into a vector search, return the top-K body strings, and let the model integrate them.

~/.claude/commands/recall.md:

---
description: Search long-term memory for facts relevant to the current task
argument-hint: <natural-language query>
---

The user wants you to consult prior memory. The block below is the top 5 results by semantic similarity, ordered most relevant first. Treat them as established context for the rest of this turn; if any contradicts the user's current message, mention the conflict before acting.

!curl -sS --fail --max-time 2 \
  -H "Authorization: Bearer $REDDB_TOKEN" \
  -H "Content-Type: application/json" \
  "$REDDB_URL/memory/search" \
  -d "$(jq -nc --arg q "$ARGUMENTS" --arg scope "$PWD" \
        '{query:$q, scope:[$scope,"global"], top_k:5}')"

The server-side SQL is the obvious thing:

SELECT id, kind, body, 1 - (embedding <=> embed($1)) AS score
FROM memory
WHERE deleted_at IS NULL
  AND scope = ANY($2)
ORDER BY embedding <=> embed($1)
LIMIT $3;

Two design notes worth lingering on:

scope = ANY($2) makes the query naturally federated. The command passes [$PWD, "global"], so project-local memories and user-global memories are pooled into one ranked list. Cross-project memory works without extra commands.
The prompt tells the model to surface contradictions. /recall is not just retrieval; it is retrieval the model is expected to reason about. A memory that says "we never use ORM migrations on this repo" should make the model push back when the user asks for a migration script, not silently obey.

`/forget` — the prune

Two shapes, depending on how the user invokes it:

/forget <id> — soft-delete a specific row by id, used after /remember returns mem_01H7….
/forget <query> — semantic forget. Find the top match, show it to the user for confirmation, only then flip deleted_at.

~/.claude/commands/forget.md:

---
description: Remove a memory by id, or find one by description and confirm before pruning
argument-hint: <id | natural-language description>
---

If `$ARGUMENTS` looks like a ulid (matches `^mem_[0-9A-Z]{26}$`), soft-delete it directly and report the id back.

Otherwise, search for the closest matching memory and show the user its body and id; do not delete until the user confirms with "yes" or "forget it".

!arg="$ARGUMENTS"; \
  if [[ "$arg" =~ ^mem_[0-9A-Z]{26}$ ]]; then \
    curl -sS --fail --max-time 2 \
      -H "Authorization: Bearer $REDDB_TOKEN" \
      -X POST "$REDDB_URL/memory/$arg/forget"; \
  else \
    curl -sS --fail --max-time 2 \
      -H "Authorization: Bearer $REDDB_TOKEN" \
      -H "Content-Type: application/json" \
      "$REDDB_URL/memory/search" \
      -d "$(jq -nc --arg q "$arg" --arg scope "$PWD" \
            '{query:$q, scope:[$scope,"global"], top_k:1}')"; \
  fi

The two-step "find, confirm, delete" flow is the part that matters. A vector search is fuzzy by design; deleting on the first match would let /forget vue preferences quietly remove a feedback row about React preferences whose embedding sat slightly closer to the query than the actual Vue one. Putting the model in the loop costs one round-trip and prevents the failure entirely.

The soft-delete itself is a one-line update on the API side:

UPDATE memory
SET deleted_at = now()
WHERE id = $1 AND deleted_at IS NULL
RETURNING id, kind, body;

RETURNING is there so the command can echo what it pruned. If you ever want an /unforget, the same row is still there — just flip deleted_at back to NULL.

State across sessions

The three commands share nothing in the harness. They share the memory table. That is what gives them coherence: writing /remember … 'use pnpm, never npm' in one session means a SessionStart hook in the next session pulls the row into the system reminder, and the model will reach for pnpm without being told twice.

The lifecycle, as a sequence:

Session A: user runs /remember --kind=feedback 'use pnpm, never npm'. Row commits.
Session A ends.
Session B boots. The SessionStart hook from D1 queries memory for top-K by scope, gets the pnpm row, injects it as context.
User in session B asks the model to install a dependency. Model uses pnpm.
Session C, six weeks later: the rule has stopped applying. User runs /forget pnpm preference. /forget finds the row, the model shows it, the user confirms. Row's deleted_at flips.
Session D: the row is no longer in the partial indexes; the model has no record of it.

Each step is independent, each step is durable, and the only thing that crosses the boundary between sessions is the table.

What this buys, in one table

Behaviour	Stateless command	Command + `memory` table
Persist a fact from a turn	impossible	`/remember`
Read prior facts in a future turn	re-read `MEMORY.md` blindly	`/recall` ranks by similarity
Prune a stale fact	edit a markdown file by hand	`/forget` with confirmation
Cross-project recall	symlink markdown files	`scope=ANY([project, "global"])`
Audit who/when/why	`git blame` if committed	`created_at` + `deleted_at` are columns

The three command files together are about 40 lines of markdown. The SQL is four statements. The state is one table. Compared to the workaround of "edit a markdown file and hope the model re-reads it," it is hard to overstate how much smaller the surface gets when the slash command is a UI over a database instead of a macro.

What is next in this pillar

An MCP server that exposes memory.write / memory.search / memory.forget as MCP tools, so every MCP-compatible CLI (Cursor, Codex, Gemini CLI) writes into the same table Claude Code's slash commands do.
A tutorial on building a full remember-this skill that captures interesting conversation snippets automatically — same table, different write path.
The context-window-economics post: how much does swapping MEMORY.md for a /recall actually save, per turn, in tokens and dollars?

If you want to be told when those land, grab the Atom feed.

Hooks that mutate state — making Claude Code hooks transactional with RedDB

2026-05-20

What a hook actually does

Claude Code fires a handful of events at the shell — SessionStart, PreToolUse, PostToolUse, Stop, a few more — and lets you bind each one to a command. The pattern most teams reach for first is an audit log: every tool call gets a row somewhere, so you can later answer "what did the agent touch on this branch?" without re-reading the transcript.

Sketched as a one-liner, the hook looks fine:

#!/usr/bin/env bash
payload="$(cat)"
curl -sS "$REDDB_URL/audit/insert" -d "$payload"

Production-fine it is not. Three failure modes:

The hook is killed mid-flight. The agent crashed, the user hit Ctrl-C, the laptop slept. The row may or may not have been written; nothing in the system knows.
The harness retries. Claude Code does not today, but a queue in front of the hook (k8s, systemd, your own wrapper) will. Two identical inserts arrive and you have phantom audit rows.
The hook fans out to a second system. Post the same audit line to Slack, write a Jira ticket, push to Datadog — and now you have a distributed transaction with no coordinator.

A single RedDB transaction solves (1) and (2). A saga solves (3). The combined recipe fits in one hook script.

The schema

One table for the audit row, one table for the idempotency keys, one table for the saga state. All three are append-only except saga.status.

CREATE TABLE audit_event (
  id            TEXT PRIMARY KEY,                -- ulid, generated client-side
  session_id    TEXT NOT NULL,
  hook_event    TEXT NOT NULL,                   -- PostToolUse, Stop, …
  tool_name     TEXT,                            -- Read, Bash, Skill, …
  tool_input    JSONB NOT NULL,
  is_error      BOOLEAN NOT NULL DEFAULT false,
  created_at    TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE TABLE hook_idempotency (
  key           TEXT PRIMARY KEY,                -- hash(session_id + tool_use_id)
  event_id      TEXT NOT NULL REFERENCES audit_event(id),
  created_at    TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE TABLE saga (
  id            TEXT PRIMARY KEY,                -- ulid
  event_id      TEXT NOT NULL REFERENCES audit_event(id),
  step          TEXT NOT NULL,                   -- 'slack', 'datadog', …
  status        TEXT NOT NULL,                   -- pending | done | failed | compensated
  attempt       INTEGER NOT NULL DEFAULT 0,
  next_run_at   TIMESTAMPTZ NOT NULL DEFAULT now(),
  payload       JSONB NOT NULL,
  last_error    TEXT
);

CREATE INDEX audit_session_idx ON audit_event (session_id, created_at DESC);
CREATE INDEX saga_pending_idx  ON saga (status, next_run_at) WHERE status = 'pending';

hook_idempotency.key is what makes the hook safe to retry. The Claude Code payload includes a tool_use_id per call; the hash of session_id + tool_use_id is unique per invocation and stable across retries. Two hook executions with the same key see a PK violation on the second insert and bail out cleanly.

The transactional insert

RedDB exposes single-statement transactions via a /tx endpoint that accepts an array of SQL statements and applies them atomically. The hook posts both rows in one call:

#!/usr/bin/env bash
# ~/.claude/hooks/audit-log.sh
set -euo pipefail

payload="$(cat)"

session_id="$(jq -r '.session_id'                <<<"$payload")"
hook_event="$(jq -r '.hook_event_name'           <<<"$payload")"
tool_use_id="$(jq -r '.tool_use_id // .id // ""' <<<"$payload")"
tool_name="$(jq  -r '.tool_name // ""'           <<<"$payload")"
tool_input="$(jq  '.tool_input // {}'            <<<"$payload")"
is_error="$(jq  -r '.is_error // false'          <<<"$payload")"

event_id="$(ulid)"
idem_key="$(printf '%s:%s' "$session_id" "$tool_use_id" | sha256sum | awk '{print $1}')"

curl -sS --fail --max-time 1 \
  -H "Authorization: Bearer $REDDB_TOKEN" \
  -H "Content-Type: application/json" \
  "$REDDB_URL/tx" \
  -d "$(jq -nc \
        --arg eid       "$event_id" \
        --arg sid       "$session_id" \
        --arg ev        "$hook_event" \
        --arg tname     "$tool_name" \
        --argjson tin   "$tool_input" \
        --argjson err   "$is_error" \
        --arg ikey      "$idem_key" \
        '{
          statements: [
            { sql: "INSERT INTO audit_event (id, session_id, hook_event, tool_name, tool_input, is_error) VALUES ($1,$2,$3,$4,$5,$6)",
              args: [$eid, $sid, $ev, $tname, $tin, $err] },
            { sql: "INSERT INTO hook_idempotency (key, event_id) VALUES ($1,$2) ON CONFLICT (key) DO NOTHING",
              args: [$ikey, $eid] }
          ]
        }')" >/dev/null || true

echo '{"continue":true}'

Three details earn their lines:

The two INSERTs share one transaction. Either both land or neither does — no orphaned audit_event row pointing at no idempotency key, and no key pointing at a non-existent event.
ON CONFLICT (key) DO NOTHING makes the hook re-entrant. A retried call inserts the audit_event row, sees the idempotency conflict, and the whole transaction rolls back. The earlier row stays as the canonical one.
--max-time 1 plus || true. Audit logging must never block the agent. If RedDB is down for a second, the hook drops that one event rather than stalling the turn. The reason this is acceptable is that the next subsection picks the dropped event back up.

If your environment cannot tolerate even a dropped event, the cure is to write to a local SQLite WAL first and let a background process drain it into RedDB. That moves the durability boundary one hop closer to the agent at the cost of a second store.

Idempotency, demonstrated

A quick way to convince yourself the recipe is correct is to invoke the hook twice with the same payload and check the count:

$ echo "$payload" | ~/.claude/hooks/audit-log.sh
$ echo "$payload" | ~/.claude/hooks/audit-log.sh

$ psql -c "SELECT count(*) FROM audit_event
           WHERE id IN (
             SELECT event_id FROM hook_idempotency
             WHERE key = '$(printf %s "$session_id:$tool_use_id" | sha256sum | awk "{print \$1}")'
           )"
 count
-------
     1

One row, not two. The second hook's transaction was rolled back by the idempotency conflict.

The saga: fanning out to external systems

The audit row is durable now. What about the Slack post that some teams want for every error tool call? You cannot put a Slack POST inside the RedDB transaction — different system, no two-phase commit. The standard answer is a saga: record the intent atomically with the audit row, run the external call separately, mark the saga step done or failed, and let a compensator deal with permanent failures.

Extend the transaction:

{
  "statements": [
    { "sql": "INSERT INTO audit_event …", "args": [/* … */] },
    { "sql": "INSERT INTO hook_idempotency …", "args": [/* … */] },
    { "sql": "INSERT INTO saga (id, event_id, step, status, payload) VALUES ($1, $2, 'slack', 'pending', $3)",
      "args": [/* saga_id */, /* event_id */, /* slack payload */] }
  ]
}

Now the audit row, the idempotency key, and the saga step land together or not at all. A separate worker drains the saga table:

-- The worker pulls one pending step at a time with row-level locking
-- so multiple workers can run safely.
BEGIN;

SELECT id, step, payload, attempt
  FROM saga
 WHERE status = 'pending'
   AND next_run_at <= now()
 ORDER BY next_run_at
 LIMIT 1
   FOR UPDATE SKIP LOCKED;

-- worker runs the step…

-- On success:
UPDATE saga SET status = 'done' WHERE id = $1;

-- On retryable failure (network blip, 5xx):
UPDATE saga
   SET attempt = attempt + 1,
       next_run_at = now() + (interval '1 second' * power(2, attempt)),
       last_error = $2
 WHERE id = $1;

-- On terminal failure (e.g. 4xx after N attempts):
UPDATE saga SET status = 'failed', last_error = $2 WHERE id = $1;

COMMIT;

FOR UPDATE SKIP LOCKED is the load-bearing piece. It turns the table into a work queue without a second piece of infrastructure. The worker holds a lock on the row it picked, every other worker skips that row, and the exponential backoff (power(2, attempt)) limits the damage a flapping downstream can do.

What about the compensator? If a Slack post fails terminally, the audit row should not be deleted — the tool call really did happen. The compensation is to write a second audit row noting the failed notification, which is itself a transactional insert:

INSERT INTO audit_event (id, session_id, hook_event, tool_name, tool_input)
VALUES (ulid(), $1, 'SagaFailed', 'slack',
        jsonb_build_object('original_event', $2, 'reason', $3));

The invariant the saga upholds is no silent loss. Either the external system gets the message, or there is a permanent record of why it did not.

What this buys you

Failure	Naïve hook	Transactional hook
Hook killed mid-write	Partial state, no record	Either fully applied or nothing at all
Harness retries	Duplicate rows	Idempotency PK rejects the second
RedDB momentarily unreachable	Lost row, no log	Lost row, logged in stderr; replay job picks it up
Slack/Jira/Datadog 503	Hook hangs the turn, or row missing	Saga step retried with backoff out-of-band
Slack 401 (permanently broken)	Silent failure	`saga.status = 'failed'` + compensating audit row

The hook script grew from one curl to about thirty lines. The schema grew by two tables. In exchange the audit log becomes the thing it claims to be — a trustworthy record of every tool call, with a clean story for every failure mode the network and the harness can throw at it.

What is next in this pillar

A /remember-shaped slash command that uses the same idempotency table so re-issuing a memory write does not create duplicate rows.
Cross-session task queues built on the same FOR UPDATE SKIP LOCKED pattern, so AFK agents can resume from a checkpoint after a crash.
An MCP server that exposes audit.* to every MCP-compatible CLI, so the same transactional guarantees apply when Cursor or Codex writes through.

The Atom feed is the way to be told when those land.

Skills as data — storing metadata, runs, and learned refinements in RedDB

2026-05-18

The flat-file problem, take two

The previous post in this pillar made the case for putting agent memory in RedDB instead of MEMORY.md. The same argument applies to skills, and the failure modes rhyme.

A Claude Code skill lives at ~/.claude/skills/<name>/SKILL.md. Its frontmatter declares a name and a description, and the harness uses the description as a coarse-grained trigger — when the user says something that matches the description, the skill is suggested. A typical install has dozens of skills (tdd, harden, optimize, caveman, huashu-design, …) and the trigger logic is plain prompt engineering: "here are the skills, here are their descriptions, pick one."

Three things that approach cannot do:

Rank by past success. If optimize produces good output 9 times out of 10 on Vue work and polish only 3 of 10, the harness has no way to know.
Compose. Two skills that frequently run in sequence (tdd → code-review-and-quality) are not linked anywhere.
Improve over time. A skill that failed because its trigger description was too vague does not get fixed by the system that observed the failure.

Treat skills as rows in a database and all three become normal queries.

Schema

Four tables. Names match what Claude Code already exposes so the mapping is obvious.

-- One row per installed skill. Mirrors the SKILL.md frontmatter.
CREATE TABLE skill (
  id            TEXT PRIMARY KEY,             -- e.g. "tdd"
  version       TEXT NOT NULL,                -- semver of the SKILL.md
  description   TEXT NOT NULL,                -- the frontmatter description
  triggers      TEXT[] NOT NULL DEFAULT '{}', -- keywords / patterns
  embedding     VECTOR(1024) NOT NULL,        -- embedded description+triggers
  installed_at  TIMESTAMPTZ NOT NULL DEFAULT now()
);

-- One row per invocation. Append-only.
CREATE TABLE skill_run (
  id          TEXT PRIMARY KEY,               -- ulid
  skill_id    TEXT NOT NULL REFERENCES skill(id),
  session_id  TEXT NOT NULL,
  args        JSONB NOT NULL,                 -- the task the skill received
  started_at  TIMESTAMPTZ NOT NULL DEFAULT now(),
  ended_at    TIMESTAMPTZ,
  outcome     TEXT,                           -- success | failed | abandoned
  notes       TEXT                            -- model's post-run reflection
);

-- Learned refinements. Updated whenever a run produces a new lesson.
CREATE TABLE skill_lesson (
  id          TEXT PRIMARY KEY,
  skill_id    TEXT NOT NULL REFERENCES skill(id),
  pattern     TEXT NOT NULL,                  -- "when args.lang=rust, prefer X"
  weight      REAL NOT NULL DEFAULT 1.0,
  embedding   VECTOR(1024) NOT NULL,
  updated_at  TIMESTAMPTZ NOT NULL DEFAULT now()
);

-- Edges between skills that ran in sequence within one session.
CREATE TABLE skill_edge (
  from_skill  TEXT NOT NULL REFERENCES skill(id),
  to_skill    TEXT NOT NULL REFERENCES skill(id),
  count       INTEGER NOT NULL DEFAULT 1,
  PRIMARY KEY (from_skill, to_skill)
);

CREATE INDEX skill_run_skill_idx ON skill_run (skill_id, started_at DESC);
CREATE INDEX skill_embedding_idx ON skill USING hnsw (embedding vector_cosine_ops);
CREATE INDEX skill_lesson_embedding_idx ON skill_lesson USING hnsw (embedding vector_cosine_ops);

The four tables cover the three things the flat-file world cannot: skill_run for ranking by outcome, skill_edge for composition, skill_lesson for refinements.

The fit query

This is where the database earns its keep. Given a new task description, return the top-K skills ranked by a blend of semantic fit and past success rate:

WITH q AS (SELECT embed($1) AS v),
stats AS (
  SELECT
    skill_id,
    count(*) FILTER (WHERE outcome = 'success')::float
      / NULLIF(count(*), 0)                              AS success_rate,
    count(*)                                             AS runs
  FROM skill_run
  WHERE started_at > now() - interval '90 days'
  GROUP BY skill_id
)
SELECT
  s.id,
  s.description,
  1 - (s.embedding <=> q.v)                              AS fit,
  COALESCE(st.success_rate, 0.5)                         AS success_rate,
  COALESCE(st.runs, 0)                                   AS runs,
  -- blend: fit dominates when runs are low, success rate matters when we have data.
  (1 - (s.embedding <=> q.v)) * 0.6
    + COALESCE(st.success_rate, 0.5) * 0.4               AS score
FROM skill s, q
LEFT JOIN stats st ON st.skill_id = s.id
ORDER BY score DESC
LIMIT $2;

Notes worth reading:

COALESCE(success_rate, 0.5) — a skill with no history gets a neutral prior, so a new skill is not punished for being new.
The 60/40 blend is a knob. If the install is brand-new, push it to 80/20 toward semantic fit. After a few hundred runs, slide it to 40/60.
interval '90 days' decays the relevance of ancient runs without explicit pruning.

Wiring it into Claude Code

The harness fires three events that are useful here: SessionStart (boot), PreToolUse/PostToolUse (every tool call, including the Skill tool), and Stop (turn end). One hook per event is enough.

1. Record installations on `SessionStart`

A short bash hook syncs installed skills into the skill table whenever a session boots. Drop it at ~/.claude/hooks/skills-sync.sh:

#!/usr/bin/env bash
set -euo pipefail

for dir in ~/.claude/skills/*/; do
  name="$(basename "$dir")"
  meta="$dir/SKILL.md"
  [[ -f "$meta" ]] || continue

  # Parse frontmatter (description, version) with yq.
  desc="$(yq -r '.description // ""' "$meta")"
  ver="$(yq -r '.version // "0.0.0"' "$meta")"

  curl -sS --fail --max-time 2 \
    -H "Authorization: Bearer $REDDB_TOKEN" \
    -H "Content-Type: application/json" \
    "$REDDB_URL/skill/upsert" \
    -d "$(jq -nc \
          --arg id "$name" \
          --arg ver "$ver" \
          --arg desc "$desc" \
          '{id:$id, version:$ver, description:$desc}')" >/dev/null
done

# Return empty hookSpecificOutput — this hook is a side effect.
echo '{"hookSpecificOutput":{"hookEventName":"SessionStart","additionalContext":""}}'

RedDB computes the embedding server-side at upsert time, so the hook stays one curl call per skill. Two seconds is enough budget for a few dozen skills; if you have hundreds, switch to a single bulk POST.

2. Log every run via `PostToolUse`

The Claude Code harness emits a PostToolUse event for every tool call. The payload includes the tool name and its arguments. Match on Skill:

{
  "hooks": {
    "PostToolUse": [
      {
        "matcher": "Skill",
        "hooks": [
          { "type": "command", "command": "~/.claude/hooks/skill-run-log.sh" }
        ]
      }
    ]
  }
}

#!/usr/bin/env bash
# ~/.claude/hooks/skill-run-log.sh
set -euo pipefail

# The harness pipes the tool-use payload as JSON on stdin.
payload="$(cat)"
skill_id="$(jq -r '.tool_input.skill' <<<"$payload")"
session_id="$(jq -r '.session_id'    <<<"$payload")"
args="$(jq    '.tool_input.args // {}' <<<"$payload")"
outcome="$(jq -r 'if .is_error then "failed" else "success" end' <<<"$payload")"

curl -sS --fail --max-time 1 \
  -H "Authorization: Bearer $REDDB_TOKEN" \
  -H "Content-Type: application/json" \
  "$REDDB_URL/skill_run/insert" \
  -d "$(jq -nc \
        --arg sid "$skill_id" \
        --arg ses "$session_id" \
        --argjson args "$args" \
        --arg outcome "$outcome" \
        '{skill_id:$sid, session_id:$ses, args:$args, outcome:$outcome}')" \
  >/dev/null || true

The trailing || true is deliberate: a logging hook must not block the agent if RedDB is briefly unreachable. The --max-time 1 budget keeps the hook invisible to the user.

3. Build skill edges in a `Stop` hook

When the turn ends, look at the ordered list of skill runs in this session and bump skill_edge.count for each adjacent pair. A single SQL statement on the API side handles the upsert:

INSERT INTO skill_edge (from_skill, to_skill, count)
VALUES ($1, $2, 1)
ON CONFLICT (from_skill, to_skill)
DO UPDATE SET count = skill_edge.count + 1;

After a week the skill_edge table is a usable Markov chain: given the user just ran tdd, what comes next? code-review-and-quality with weight 0.6, git-workflow-and-versioning with 0.3.

A `/suggest-skill` slash command

Put the fit query behind a slash command and the model can call it directly:

~/.claude/commands/suggest-skill.md:

---
description: Suggest the best-fitting skills for a task using past run data
argument-hint: <task description...>
---

Here are the top 5 skills ranked by semantic fit blended with 90-day success rate. Pick one and invoke it — do not ask the user to confirm if the top score is above 0.75.

!curl -sS --fail \
  -H "Authorization: Bearer $REDDB_TOKEN" \
  -H "Content-Type: application/json" \
  "$REDDB_URL/skill/suggest" \
  -d "$(jq -nc --arg q "$*" '{query:$q, top_k:5}')"

Usage in a session:

/suggest-skill the alignment between cards in the pricing grid feels off on mobile

The model gets a JSON list back, reads the top entry (probably polish or frontend-design), and invokes it without a round-trip. The keyword "alignment" did not have to be in any skill's static description — the embedding does the work.

Learned refinements

Skills get better when the system that runs them remembers which patterns worked. Add a Stop hook that asks the model itself to write a one-line lesson when a run was particularly good or bad:

# Pseudo: read last skill_run for this session, if outcome=success
# and notes is non-empty, write a skill_lesson row.
curl -sS \
  -H "Authorization: Bearer $REDDB_TOKEN" \
  -H "Content-Type: application/json" \
  "$REDDB_URL/skill_lesson/upsert" \
  -d "$(jq -nc \
        --arg sid "$skill_id" \
        --arg pat "$lesson" \
        '{skill_id:$sid, pattern:$pat}')"

At suggestion time, join skill_lesson into the fit query and surface the top-1 lesson alongside the skill name:

SELECT s.id,
       s.description,
       (SELECT pattern FROM skill_lesson l
        WHERE l.skill_id = s.id
        ORDER BY l.embedding <=> q.v ASC
        LIMIT 1)                                       AS hint,
       …
FROM skill s, q
…

Now /suggest-skill returns not just which skill, but which lesson applies. A skill that learned "when args.lang=rust prefer cargo nextest over cargo test" feeds that hint back to the model before it invokes.

What you get

Question	Flat-file world	Skills-as-data
Which skill fits this task?	string match on description	vector search blended with success rate
Which skills compose well?	folklore	`skill_edge` Markov chain
Which skill is unreliable lately?	feel	`success_rate < 0.5 AND runs > 20`
Why did `optimize` fail last Tuesday?	not recorded	`SELECT … FROM skill_run WHERE …`

Total wiring: four tables, three hooks, one slash command, one API. The skills themselves do not change — they are still SKILL.md files on disk. The harness still reads them. The database sits beside the harness and turns every invocation into a row, every row into ranking signal, and every ranking signal into a better suggestion the next time around.

What is next in this pillar

A walkthrough of building a remember-this skill end-to-end in 30 minutes — same shape, applied to conversation snippets.
Hooks that mutate persistent state, made transactional so a half-finished skill_run insert does not leave you with a phantom row when the agent crashes mid-turn.
An MCP server that exposes skill.* and memory.* to every MCP-compatible CLI, so Cursor and Codex can read the same tables Claude Code writes.

If you want to be told when those land, grab the Atom feed.

Agent SDK on RedDB: building custom agents with native persistence

2026-05-17

If you have built an agent on top of a hosted CLI like Claude Code, you have probably hit the wall where the wrapper stops fitting. The conversation log is somebody else's schema. The tool registry is a JSON file in a config directory. The memory is a markdown file the host parses on startup. The eval hooks, if they exist, run after the fact and write to a separate store. It works until you want to ask "what is the second-most-common false positive my review bot has produced on this repo over the last 90 days," at which point you discover the data lives in four places and none of them join.

This post is about the alternative: a small, opinionated Agent SDK whose four surfaces all sit on a single RedDB instance. Not a framework — there is no scheduler, no agent-of-agents abstraction, no DSL. Just the four pieces every agent project ends up reimplementing badly, given names and a storage contract. The point is that once those four surfaces share a database, the queries that used to require glue code become one statement.

We end with a code-review bot. Under 300 lines, including the SDK plumbing, that remembers every review it has ever produced per repository, learns from human overrides, and lets you ask "what did I get wrong" in SQL.

The four surfaces

The SDK exposes exactly four things to the agent author. Adding a fifth is the moment you should reach for a framework instead.

Tool registry. A typed catalogue of the tools the agent can call. Tools are rows, not files. Each row stores the JSON schema for arguments, the handler reference, the rate-limit budget, and a hash of the implementation so the eval layer can tell when a tool's behaviour changed underneath an old eval run.

Memory adapter. The agent's read/write surface for facts it wants to remember. The SDK does not assume a single shape of memory — episodic, semantic, and procedural memory each have their own table, with the access patterns described in Building an agent memory layer on RedDB. The adapter is a thin facade that hides which table a remember(...) call lands in, based on the kind of fact.

Conversation log. Append-only stream of every turn: user input, model output, tool call, tool result, error. Indexed by conversation id, by time, and (for the entries with embedded content) by vector. The log is the source of truth — everything else, including the memory tables, is materialised from it.

Eval hooks. Two callbacks: onTurnComplete(turn) and onConversationComplete(conv). The SDK does not run evals; it gives the eval author a deterministic place to write feedback rows that join back to the conversation. Evals run as separate workers reading from the same database. There is no "send results to the eval service" step — the results are already there.

Four surfaces, one database, no service mesh.

The storage contract

The SDK ships with a migration that creates five tables. Names matter — the queries in the rest of this post and in the code-review bot below reference them.

-- 1. The conversation log. Append-only.
CREATE TABLE agent_turns (
  id            uuid PRIMARY KEY,
  conversation  uuid NOT NULL,
  agent         text NOT NULL,
  kind          text NOT NULL,  -- 'user' | 'model' | 'tool_call' | 'tool_result' | 'error'
  payload       jsonb NOT NULL,
  embedding     vector(1024),
  created_at    timestamptz NOT NULL DEFAULT now()
);

CREATE INDEX agent_turns_conv_time ON agent_turns (conversation, created_at);
CREATE INDEX agent_turns_embedding ON agent_turns USING hnsw (embedding vector_cosine_ops);

-- 2. The tool registry.
CREATE TABLE agent_tools (
  name          text PRIMARY KEY,
  agent         text NOT NULL,
  schema        jsonb NOT NULL,
  impl_hash     text NOT NULL,
  rate_budget   jsonb NOT NULL,  -- { window_seconds, max_calls }
  updated_at    timestamptz NOT NULL DEFAULT now()
);

-- 3. Memory: episodic, semantic, procedural.
CREATE TABLE agent_episodic  ( /* see agent-memory-layer-on-reddb */ );
CREATE TABLE agent_semantic  ( /* see agent-memory-layer-on-reddb */ );
CREATE TABLE agent_procedural( /* see agent-memory-layer-on-reddb */ );

-- 4. Eval feedback. Joins back to a specific turn.
CREATE TABLE agent_evals (
  id            uuid PRIMARY KEY,
  turn_id       uuid NOT NULL REFERENCES agent_turns (id) ON DELETE CASCADE,
  evaluator     text NOT NULL,    -- 'human', 'rubric:v3', 'llm-judge:gpt-4o'
  verdict       text NOT NULL,    -- 'good' | 'bad' | 'partial'
  rationale     text,
  payload       jsonb,
  created_at    timestamptz NOT NULL DEFAULT now()
);

CREATE INDEX agent_evals_turn ON agent_evals (turn_id);

The five tables are the entirety of the SDK's storage surface. Everything else — replay, ranking, "what reviews has this bot done on this repo," "which tool calls produced bad outcomes last week" — is one or two joins on this schema. The shape is deliberately boring.

A few decisions worth pointing at:

agent_turns.embedding is on the turn row, not in a sidecar table. The same write that logs the model's output also writes its embedding, in one transaction. There is no two-step "log then embed" that can fail halfway.
agent_tools.impl_hash exists so the eval layer can refuse to compare a current verdict against a historical one when the tool implementation has changed. Without it, eval drift is silent.
agent_evals.turn_id is a foreign key with ON DELETE CASCADE. A retention policy on agent_turns automatically prunes the evals that referenced them, so you do not end up with orphan rows pointing at conversations that have aged out.

The SDK shape, in TypeScript

The code below is the entire user-facing surface. It is small on purpose; the value is in the storage contract above, not in the wrapper.

import { RedDB } from '@reddb/client';
import { embed } from '@reddb/embed';

export type Tool<A, R> = {
  name: string;
  schema: object;                       // JSON Schema for `args`
  handler: (args: A, ctx: Ctx) => Promise<R>;
  rateBudget?: { windowSeconds: number; maxCalls: number };
};

export type EvalHooks = {
  onTurnComplete?: (turn: Turn) => Promise<void>;
  onConversationComplete?: (conv: Conversation) => Promise<void>;
};

export class Agent {
  constructor(
    private db: RedDB,
    private name: string,
    private tools: Tool<any, any>[],
    private hooks: EvalHooks = {},
  ) {}

  async register() {
    await this.db.tx(async (tx) => {
      for (const t of this.tools) {
        await tx.upsert('agent_tools', {
          name: t.name,
          agent: this.name,
          schema: t.schema,
          impl_hash: hashFn(t.handler),
          rate_budget: t.rateBudget ?? { windowSeconds: 60, maxCalls: 30 },
        });
      }
    });
  }

  async run(conversation: string, userInput: string, ctx: Ctx): Promise<string> {
    // Append user turn (with embedding) in one transaction.
    const userEmb = await embed(userInput);
    await this.db.insert('agent_turns', {
      conversation, agent: this.name, kind: 'user',
      payload: { text: userInput }, embedding: userEmb,
    });

    // Hand the conversation to the model. Loop until the model stops
    // requesting tools. Each tool call and result is its own logged turn.
    let modelOut: ModelTurn;
    while (true) {
      modelOut = await callModel(this.db, this.name, conversation, this.tools);
      const t = await this.logTurn(conversation, 'model', modelOut);
      await this.hooks.onTurnComplete?.(t);

      if (!modelOut.toolCall) break;

      const tool = this.tools.find((x) => x.name === modelOut.toolCall!.name);
      if (!tool) throw new Error(`unknown tool: ${modelOut.toolCall.name}`);

      await this.checkRateBudget(tool);
      const result = await tool.handler(modelOut.toolCall.args, ctx);

      const callTurn   = await this.logTurn(conversation, 'tool_call',   modelOut.toolCall);
      const resultTurn = await this.logTurn(conversation, 'tool_result', result);
      await this.hooks.onTurnComplete?.(callTurn);
      await this.hooks.onTurnComplete?.(resultTurn);
    }

    await this.hooks.onConversationComplete?.(
      await this.loadConversation(conversation),
    );
    return modelOut.text ?? '';
  }

  // ...logTurn, loadConversation, checkRateBudget elided; ~40 LOC each
}

That is the whole SDK. register() writes the tool catalogue. run() walks one turn through the model, calling tools until the model stops asking. Every turn is logged with its embedding in the same write, and the eval hooks fire at the boundaries the eval layer cares about. There is no scheduler, no retry policy, no per-tool dependency injection — those belong to the agent author, not the framework.

The memory adapter is intentionally not in the class. It is a separate helper the tool handlers use directly:

export const memory = {
  remember:   (db: RedDB, kind: 'episodic'|'semantic'|'procedural', row: object) => ...,
  recall:     (db: RedDB, kind: ..., query: { text?: string; subjects?: string[] }) => ...,
  forget:     (db: RedDB, kind: ..., predicate: object) => ...,
};

Tools write to memory when they have something worth remembering. The SDK does not guess for them.

A code-review bot, end to end

The bot is one file. It reads a pull request, walks the diff hunk by hunk, asks the model for review comments on each hunk, posts them, and — the part that matters — remembers what it has reviewed per repository so it does not produce the same comment twice and so the human override signal feeds back into future reviews.

// review-bot.ts — ~250 LOC including the SDK calls above
import { Agent, memory } from './sdk';
import { RedDB } from '@reddb/client';
import { github } from './github';   // small wrapper around @octokit/rest

const db = new RedDB(process.env.REDDB_URL!);

const tools = [
  {
    name: 'fetch_pr_diff',
    schema: { type: 'object', properties: { repo: {type:'string'}, pr: {type:'number'} }, required:['repo','pr'] },
    handler: async ({ repo, pr }, ctx) => {
      const cached = await memory.recall(db, 'procedural', { subjects: [`gh:${repo}#${pr}`] });
      if (cached.length) return cached[0].value;

      const diff = await github.diff(repo, pr);
      await memory.remember(db, 'procedural', {
        cache_key: `gh:${repo}#${pr}`, value: diff, expires_at: in1h(),
      });
      return diff;
    },
  },
  {
    name: 'prior_findings',
    schema: { type: 'object', properties: { repo:{type:'string'}, hunk_hash:{type:'string'} }, required:['repo','hunk_hash'] },
    // The whole point of the bot: do not repeat yourself.
    handler: async ({ repo, hunk_hash }) => {
      return await memory.recall(db, 'semantic', {
        subjects: [`repo:${repo}`, `hunk:${hunk_hash}`],
      });
    },
  },
  {
    name: 'post_review_comment',
    schema: { type: 'object', properties: {
      repo:{type:'string'}, pr:{type:'number'}, path:{type:'string'},
      line:{type:'number'}, body:{type:'string'}, hunk_hash:{type:'string'},
    }, required: ['repo','pr','path','line','body','hunk_hash'] },
    handler: async (a) => {
      const url = await github.postReviewComment(a);
      // Remember the finding so prior_findings will surface it next time.
      await memory.remember(db, 'semantic', {
        subjects: [`repo:${a.repo}`, `hunk:${a.hunk_hash}`, `pr:${a.pr}`],
        text: a.body, attributes: { path: a.path, line: a.line, url },
      });
      return { url };
    },
  },
];

const hooks = {
  // When a human resolves or dismisses a comment, write an eval row so
  // future runs of the bot can see which findings were accepted.
  onTurnComplete: async (turn) => {
    if (turn.kind !== 'tool_result') return;
    if (turn.payload?.toolName !== 'post_review_comment') return;
    const override = await github.checkComment(turn.payload.url);
    if (override?.dismissedBy) {
      await db.insert('agent_evals', {
        turn_id: turn.id,
        evaluator: 'human',
        verdict: 'bad',
        rationale: override.reason ?? null,
      });
    }
  },
};

const agent = new Agent(db, 'code-reviewer', tools, hooks);
await agent.register();

export async function review(repo: string, pr: number) {
  const conv = newUuid();
  return agent.run(conv, `Review PR ${repo}#${pr}. Skip findings that match prior accepted findings on the same hunk.`, { repo, pr });
}

The body of the bot is small because everything that would normally be glue — caching the diff, storing findings, recording human overrides, joining all three for the next run — is just a table write. The line that matters is the one in prior_findings: every review the bot has ever posted on this repository, for this hunk, is one recall away. The bot can decline to repeat itself without anyone writing a deduplication service.

Asking "what did I get wrong"

This is the query that motivated the whole exercise. Given the schema above, the answer is two joins:

SELECT
  t.payload ->> 'body'      AS comment_body,
  t.payload ->> 'path'      AS path,
  count(*)                  AS times_dismissed,
  max(e.rationale)          AS last_reason
FROM agent_turns t
JOIN agent_evals e ON e.turn_id = t.id
WHERE t.agent = 'code-reviewer'
  AND t.kind  = 'tool_result'
  AND t.payload ->> 'toolName' = 'post_review_comment'
  AND e.verdict = 'bad'
  AND t.payload ->> 'repo' = $1
  AND t.created_at > now() - interval '90 days'
GROUP BY 1, 2
ORDER BY times_dismissed DESC
LIMIT 20;

The most-dismissed comments per repo over the last quarter. No exporters, no warehouse, no scheduled job that flattens the data into a third store. The conversation log is the source of truth and the analytics surface, because they are the same table.

Cross-conversation learning

The second query the bot benefits from is one the SDK enables for free. "Find a review I have written on this repo whose target hunk is semantically similar to the one I am about to review, and which a human accepted." That is hybrid search on agent_turns joined to agent_evals:

SELECT t.payload, 1 - (t.embedding <=> $1) AS sim
FROM agent_turns t
LEFT JOIN agent_evals e ON e.turn_id = t.id
WHERE t.agent = 'code-reviewer'
  AND t.payload ->> 'repo' = $2
  AND (e.verdict IS NULL OR e.verdict <> 'bad')
ORDER BY t.embedding <=> $1
LIMIT 10;

The bot can prefer past wording that survived human review and avoid wording that did not. Without the embedding on the turn row, this query is a fan-out to a separate vector store with a separate consistency story; with it, it is a single planner call. The shape of the query is the planner walkthrough from Hybrid search done right applied to this exact use case.

What you give up

The SDK is small because it leaves a lot to the author. That is the trade.

No retry policy. If a tool's network call fails, the handler decides whether to retry. The SDK logs every attempt as its own tool_call / tool_result turn, so observability is preserved, but it will not retry on your behalf.
No scheduler. One run() call equals one conversation. Concurrency and queuing are outside the SDK. (For agents that need a queue, see Sub-agent dispatch queue.)
No prompt templates. callModel is a leaf the author owns. The SDK does not impose a prompt structure; it imposes a storage structure.
No multi-agent coordination. Two agents talking to each other write to the same agent_turns table under different agent values, but the SDK does not route messages between them. That is the topic of Multi-agent shared memory.

These are deliberate omissions. Every one of them is something teams want to customise per project, and every one of them is something frameworks get wrong by trying to standardise.

Why this works on RedDB specifically

Every claim in this post — embeddings on the turn row, hybrid lexical+vector retrieval against the log, foreign-keyed eval rows that survive retention pruning, one-statement analytics across log and evals — depends on three database properties: a single transaction can write a row and its embedding; the query planner can mix lexical, vector, and structured predicates in one plan; and the same instance is the operational store and the analytics surface.

You can build the same SDK on a stack of Postgres + Redis + a vector service + a warehouse. People do. The cost is the integration code that keeps the four stores in agreement, and the queries you no longer write because they would require denormalising across all of them. The SDK above is small because that integration code does not exist.

If you want to try this against your own agent, the migration and the SDK scaffold above are enough to start. Point a fresh RedDB at it, swap in your model call, and the first run() will land a row in agent_turns that you can query in the same shell you are running the bot from. The five-table shape is the contract. Everything else is yours.

Chunking inside the engine: when the DB owns segmentation

2026-05-17

The standard RAG stack chunks documents in application code. A worker reads a row from Postgres, runs it through a chunker — recursive character splitter, semantic chunker, whatever this quarter's library prefers — embeds each chunk, and writes the results to the vector store. The chunker is two hundred lines of Python that nobody on the team wrote, that everyone is afraid to touch, and that disagrees in subtle ways with whatever the next service in the pipeline expects.

This post is about what happens when that chunker moves into the database. Not as a stored procedure that imitates the Python — as a declarative rule attached to the column it segments. The rule runs inside the same transaction as the write, the chunks are queryable rows, and reranking — the other half of retrieval that usually lives in application code — becomes a query operator the planner can reason about. The comparison point is the same one we keep coming back to: an external pipeline that you have to keep in sync with the data, versus a property of the column itself.

What "chunking in app code" actually costs

If you have built a RAG pipeline, the diagram is familiar. A document arrives. A worker picks it up, applies a chunking function, embeds each chunk, writes the chunks to a vector store, optionally writes back a denormalised pointer table to the source database so you can join chunks back to documents. At query time, the application embeds the question, asks the vector store for top-k chunks, fetches the source documents from the primary store, optionally reranks the chunks with a second model, and returns the result.

Every arrow in that pipeline is a place chunking can disagree with the data:

The document changes; the chunks have not been recomputed yet. The retriever returns chunks whose source paragraph no longer exists. We covered this drift window in a previous post — chunking is where the window is widest, because chunking is the most expensive step in the pipeline and therefore the most likely to lag.
The chunking function changes — a new splitter, a new overlap setting, a new maximum size. Now half the corpus is chunked one way and half is chunked another way, and the retrieved top-k mixes the two with no way to tell which is which.
The chunking function and the retrieval code disagree about how to reassemble chunks back into context. The retriever returns chunk 5 of document 17; the assembler asks for "the paragraph around chunk 5"; the chunker's notion of "around" was never written down anywhere because it was implicit in the Python.

Each of these is a class of failure that the application is structurally bad at preventing, because the application does not own the storage and does not see the writes. The database does.

Declarative chunking, attached to the column

The mental shift is the same one we made for embeddings. Stop thinking of "the chunks for document 17" as a separate object that has to be kept in sync. Start thinking of chunking as a property of the column the chunks come from — a derived view that the engine maintains automatically, the same way it maintains an index.

The shape is a CHUNK BY clause on the column definition:

CREATE TABLE documents (
  id          UUID PRIMARY KEY,
  tenant_id   UUID NOT NULL,
  title       TEXT NOT NULL,
  body        TEXT NOT NULL CHUNK BY (
    strategy  => 'recursive',
    max_tokens => 512,
    overlap    => 64,
    separators => ARRAY[E'\n\n', E'\n', '. ', ' '],
    embed_with => 'text-embedding-3-large'
  ),
  updated_at  TIMESTAMPTZ NOT NULL DEFAULT now()
);

The CHUNK BY clause is the contract. The engine guarantees that for every committed row of documents, there exists a corresponding set of rows in a derived documents__body_chunks relation, each with the segmented text and its embedding, and that the set is consistent with the body value at the row's current visible version. There is no worker, no queue, no outbox. The chunking and embedding happen inline with the write — or, for large documents, asynchronously with the row marked chunks_pending = true until the derived relation catches up, in which case the engine refuses to return that row from a retrieval query that requires chunks.

The derived relation is a first-class table you can query directly:

SELECT c.ordinal, c.text, c.embedding
FROM documents__body_chunks c
WHERE c.document_id = $1
ORDER BY c.ordinal;

It has its own HNSW index built and maintained by the engine. It has a foreign key to the parent document that the engine cannot let you violate. Deleting the parent deletes the chunks. Rewriting body re-runs the chunker and re-embeds, all inside the writing transaction, all visible atomically to the next reader.

The point is not that the syntax is shorter. The point is that the chunking function is part of the schema, which means it is part of EXPLAIN, it is part of migrations, it is part of the backup, and most importantly it is part of the contract the engine enforces. A row of documents whose body has been written but whose chunks have not yet been generated is a row in a defined state (chunks_pending) with defined visibility rules. It is not "a bug we will find in three weeks when a customer reports stale retrieval".

What "consistent with the current value" actually means

The hardest part of chunking-as-derived-data is not running the chunker. It is the invariant that the chunks for row 17 always reflect the current committed body of row 17, for every transactional reader, under concurrent writes.

The engine handles this the way it handles any derived state: the chunks live under the same MVCC regime as the row they derive from. When a transaction writes a new body, it also writes the new chunk set, both tagged with the same commit LSN. Readers at an earlier snapshot see the old body and the old chunks. Readers at the new snapshot see the new body and the new chunks. There is no window — not zero-with-an-asterisk, just zero — in which a retrieval query can join the new body to old chunks or vice versa. This falls out of having a single WAL across data models: the document write, the chunk inserts, the chunk deletes, and the vector index updates all serialise through the same ordered byte stream.

For large bodies the chunking and embedding are too expensive to run inline. The engine has two modes:

CHUNK BY (..., commit_mode => 'inline') — chunking runs inside the writing transaction. The transaction does not commit until the chunks are computed and indexed. Visibility is bound to the row's commit. Latency is bounded by the embedder.
CHUNK BY (..., commit_mode => 'async') — the row commits with chunks_pending = true. Chunking runs in a background lane backed by the same WAL. Until it completes, retrieval queries that require chunks treat the row as not-yet-visible (they do not return stale chunks from the previous version of the row). The default for columns wider than a configurable threshold.

async is the realistic default for production. The guarantee it gives is not "the chunks are always up to date" — that is a lie any architecture would have to tell — but rather "either you see the new row with its new chunks, or you see the old row with its old chunks, never a mix." That is the only guarantee an application can actually program against.

Reranking as a query operator

Chunking is half of the retrieval story. The other half is reranking — the second-pass scoring that takes the top-N candidates from the first-pass retriever and re-orders them with a more expensive model. In a stitched stack, reranking is application code: fetch top-50 from the vector store, send them to the rerank API, sort by the new scores, take the top-5.

Once chunks are rows in the engine, reranking can be an operator the planner schedules:

SELECT id, document_id, text, score
FROM documents__body_chunks
WHERE tenant_id = $1
ORDER BY rerank(
  query    => $2,
  text     => text,
  model    => 'bge-reranker-v2',
  first_pass => hybrid(
    lexical => bm25(text, $2),
    vector  => 1 - (embedding <=> $3),
    weights => (0.4, 0.6)
  ),
  first_k  => 50
) DESC
LIMIT 10;

rerank() is recognised by the planner the same way hybrid() is — see the hybrid search post for the underlying machinery. The planner treats it as a two-stage top-k: the inner first_pass produces a candidate set of size first_k using the cheap hybrid score; the outer reranker is called once per candidate to produce the final score; the operator returns the top LIMIT by reranked score.

EXPLAIN on that statement makes the staging visible:

Limit  (cost=2104.50..2104.55 rows=10 width=160) (actual rows=10 loops=1)
  ->  Rerank Top-K  k=10  model=bge-reranker-v2
        first_k: 50  (capped from 100 by rerank_budget_ms=200)
        Per-candidate cost: 3.8 ms (calibrated)
        ->  Hybrid Top-K  k=50  weights=(0.40, 0.60)
              ... (see hybrid-search post)
        Reranked: 50 candidates, 10 returned
        Budget used: 190 ms of 200 ms
 Planning Time: 0.42 ms
 Execution Time: 214.6 ms

The planner pieces that matter:

first_k is chosen by the planner, not the application. A larger first_k improves the chance the right answer is in the candidate set but spends more reranker calls. The planner picks the largest first_k that fits the per-query rerank_budget_ms, given the calibrated per-candidate cost of the chosen reranker. The application sets a latency budget; the engine spends it.
The reranker model is named in the query and resolved like any other dependency. Models are registered in a rerankers catalog (same shape as the embedders catalog) with their endpoint, calibrated latency, and pricing. Schema migrations against the model version are the same machinery as any other schema migration.
The first-pass scoring is reused, not recomputed. Both the lexical and vector sub-scores from the hybrid() first pass are carried on the candidate rows. The reranker sees them as features it can use (bge-reranker-v2 ignores them; some rerankers consume them as a prior). Either way, no work is repeated.

The application-side equivalent of all this is the recurring bug where the candidate-set size is hard-coded to 50, the reranker takes 6 ms per call instead of the 3 ms it took six months ago, the P95 latency on the retrieval endpoint quietly drifts past the SLO, and nobody notices until a sales engineer complains. With the operator, the budget is the configured value, and the planner adjusts first_k to fit. The number tuned for last quarter's reranker is not in your code at all.

What changes for the application

Two things that used to be the application's job are now the engine's.

Chunking is no longer an out-of-band pipeline. The chunking function is a column property. Re-chunking after a strategy change is a schema migration: ALTER COLUMN body CHUNK BY (...) re-runs the chunker against every existing row, in the background, against the same WAL, with the same MVCC guarantees as any other migration. The day you change max_tokens from 512 to 768 is not the day you stand up a parallel pipeline and forklift. It is the day you write one DDL statement and watch a progress counter.

Reranking is no longer two round trips and a sort in application code. The reranker is named in the query, the budget is named in the configuration, and the candidate-set size is something the planner chooses. The half-page of TypeScript that used to call the rerank API and resort the results — gone. So is the entire class of bug where the resort used the wrong score field.

What you give up: control. If your chunker is doing something genuinely custom — semantic chunking by paragraph topic, code-aware chunking by AST boundaries, layout-aware chunking from PDF tokens — you cannot express it as strategy => 'recursive'. The escape hatch is to register a chunker as a UDF and use strategy => 'udf:my_chunker', but UDF chunkers run in a sandbox with bounded CPU and memory and cannot make network calls. For most teams the four built-in strategies (fixed, recursive, sentence, markdown) cover the corpus; for teams that need more, the UDF escape hatch exists but the bounds are real.

When you would still pick the stitched stack

There are real cases where moving chunking into the engine is the wrong call.

Your chunker calls a remote service per chunk. A document-AI service that returns layout-aware spans, an LLM that produces summary chunks, anything that makes a network hop per chunk — these cannot run inside a database transaction without inverting your latency profile. Keep the chunker external, treat its output as authoritative, write the chunks as plain rows.
You re-chunk constantly as part of an experiment. If you are sweeping chunking strategies as part of a retrieval-eval loop, you want chunking to be a function you call from a notebook, not a column property you migrate. The engine's path is for the chunker you have decided on, not the one you are still searching for.
The chunker is the product. If your differentiation is a proprietary chunking strategy you ship as a library, the strategy belongs in your code, not in your customer's database. The engine path is for teams whose differentiation is what they do with the chunks, not how they make them.

For everything else — the long tail of "we just need recursive 512/64 chunks with cosine recall and a reranker" — the stitched stack is paying ongoing operational cost for flexibility nobody is using. The schema column gives you the same chunks with no pipeline, no drift, and a query planner that can see what you are asking for.

That is the whole pitch. Chunking is a property of the column it segments. Reranking is an operator the planner schedules against a budget. The Python glue layer that used to live between Postgres and a vector store does not get smaller — it does not exist.

Hybrid search done right: lexical, vector, and filter in one plan

2026-05-17

If you have ever wired up "hybrid search" by hand, you know the shape: send the same query to a full-text index, send it again to a vector index, post-process the two result sets in application code, then filter the merged list against your structured predicates. It works. It is also three round trips, two scoring functions that disagree about what "relevant" means, and a filter step that runs after the top-k has already been chosen — which is the wrong order if your filter is even slightly selective.

This post is about how the RedDB query planner avoids that shape. Lexical, vector, and filter are not three pipelines glued together at the application layer; they are three access paths the optimizer can mix inside one plan. The interesting part is not that the syntax is shorter. The interesting part is what the planner does when one of the three legs is degenerate — when the lexical match is empty, when the vector recall is low, when the filter is selective enough to dominate the cost model. That is where the naive stitched-together version silently returns the wrong rows.

The query, as written

The user-facing API is one statement. A tenant-scoped retrieval for a customer support agent, matching on the question text both lexically and semantically, restricted to documents the asking user is allowed to see and to a recency window:

SELECT id, document_id, body, score
FROM chunks
WHERE tenant_id = $1
  AND acl_subjects && $2          -- array overlap with the caller's groups
  AND created_at > now() - interval '180 days'
  AND match(body, $3)             -- BM25 lexical predicate
ORDER BY
  hybrid(
    lexical => bm25(body, $3),
    vector  => 1 - (embedding <=> $4),
    weights => (0.4, 0.6)
  ) DESC
LIMIT 20;

$3 is the query string; $4 is its embedding. hybrid() is not a UDF that runs after the rows have been collected — it is a planner-recognised scoring construct, the same way ORDER BY ... LIMIT k over a vector distance is recognised as a top-k operator rather than a sort. The distinction matters; we will come back to it.

What the planner actually builds

Run EXPLAIN (analyze, costs, format text) on the statement above against a 24M-chunk corpus and you get something close to this:

Limit  (cost=1842.10..1842.15 rows=20 width=128) (actual rows=20 loops=1)
  ->  Hybrid Top-K  k=20  weights=(0.40, 0.60)
        Score: 0.40 * bm25(body, $3) + 0.60 * (1 - cosine(embedding, $4))
        ->  Filtered Candidate Stream  (predicate-pushdown=true)
              Filter: tenant_id = $1
                      AND acl_subjects && $2
                      AND created_at > '2025-11-18 00:00:00'
              ->  Bitmap-AND
                    ->  Lexical Index Scan  on chunks_body_fts
                          Recheck Cond: match(body, $3)
                          Heap Blocks: exact=4112
                          Rows: 38,221
                    ->  Vector ANN Scan  on chunks_embedding_hnsw
                          ef_search: 96 (auto-tuned for k=20)
                          Probe Rows: 240
                          Rows: 218
              Bitmap-AND output: 412 candidate rows
        Rescored: 412 rows  (lexical + vector, with stored sub-scores)
        Buffers: shared hit=1894
 Planning Time: 0.31 ms
 Execution Time: 11.4 ms

Three things are worth pointing at.

First, the filter is pushed down to candidate generation, not applied after the top-k. The lexical scan and the ANN scan each emit row identifiers; the structured filter is evaluated as those identifiers stream out of the indexes, before the bitmap is intersected. If tenant_id = $1 cuts the corpus from 24M to 80k chunks, the ANN scan only walks the part of the HNSW graph that points into that tenant's slice — the index stores a per-row tenant id in the leaf payload so the graph traversal can skip out-of-tenant nodes without a separate lookup.

Second, the two candidate streams are intersected, not unioned. The naive recipe — "take top-100 lexical, take top-100 vector, merge by some weighted score" — unions and reranks. That is the wrong default for a system where both signals are meant to point at the same row. A union biases towards documents that are strong on one signal and absent on the other, which is exactly the kind of result a human reviewing the output calls "irrelevant but on-topic". The planner intersects by default and falls back to a union only when the intersection is too small to fill the requested k (see the edge cases below).

Third, scoring happens once, on the surviving candidates. bm25(body, $3) is computed during the lexical index scan and the sub-score is carried on the candidate row; cosine(embedding, $4) is computed during the ANN scan and similarly carried. The Hybrid Top-K operator combines the two stored sub-scores with the requested weights and runs a bounded heap of size k. There is no recompute, no second pass over the corpus, and no application-layer merge.

The cost model is what makes this stable under changing data shape. The planner estimates the selectivity of each leg — filter, lexical, vector — and chooses both the join order of the bitmaps and the ef_search parameter for the ANN scan to satisfy a target recall. It is the same machinery that picks an index in a normal OLTP planner, with two additions: an approximate-recall estimator for the ANN side, and a Zipfian estimator for term frequency on the lexical side.

What changes when a leg is degenerate

The interesting failure modes of hybrid search are not "everything works and the score is slightly off". They are the cases where one of the three legs collapses. The planner has a defined behaviour for each.

Empty lexical match

The user asks something the corpus has no lexical overlap with. Common when the query is a paraphrase, an acronym not present in the documents, or a different language than the indexed text. The lexical leaf returns zero rows.

A naive merge would now return whatever the vector leg found, ranked purely by cosine — which is fine, except that the weighted score 0.4 * bm25 + 0.6 * cosine collapses to 0.6 * cosine for every row, and you have silently changed your ranking function in the worst possible way: the API still claims to be hybrid.

The planner handles this explicitly. When the lexical bitmap is empty and the vector bitmap is non-empty, the Hybrid Top-K operator switches to the vector-only score and emits a runtime annotation on the result set:

Hybrid Top-K  k=20  weights=(0.40, 0.60)
  Degraded: lexical_empty -> vector_only
  Rescored: 218 rows  (vector sub-score only; lexical sub-score = 0)

The application sees a degraded field on the response envelope ({ degraded: "lexical_empty" }). It is not an error — the rows are still useful — but the caller now has the signal it needs to, for example, prompt the agent for a clarifying question, or to suppress the citation footnotes that pretend the lexical signal participated.

Low-recall vector match

Pathological the other direction: the ANN scan returns very few candidates, either because the embedding is far from any cluster the index has visited at the current ef_search, or because the filter eliminated most of the ANN's neighbourhood. The planner notices the candidate count is below a configurable floor (hybrid.vector_recall_floor, default 0.5 * k) and re-runs the ANN scan with ef_search * 2, up to a configurable ceiling. This is cheap because the HNSW state from the first pass is cached; the second pass continues the traversal from where the first stopped, rather than starting from the entry point.

If after the ceiling the vector candidate count is still below the floor, the operator widens the candidate set by unioning the lexical bitmap rather than intersecting — and again annotates the result:

Hybrid Top-K  k=20  weights=(0.40, 0.60)
  Degraded: low_vector_recall -> union(lexical, vector)
  ef_search escalation: 96 -> 192 -> 384 (ceiling)
  Rescored: 184 rows

Note that the score formula is unchanged — both sub-scores are still combined with the original weights — but the candidate set is now a union, which is the right default when the planner knows the intersection is artificially small.

Filter-dominant queries

The third edge case is the one that quietly destroys naive stacks. The filter is highly selective: a tenant with 200 chunks, or a date window that excludes 99% of the corpus. A bolt-on hybrid stack first asks the lexical index for top-100 and the vector index for top-100, then filters down — and ends up with a handful of rows, or zero, despite the underlying tenant slice containing perfectly good matches.

The planner detects this from the filter's estimated selectivity. When the filter cardinality is below a threshold relative to k (default: filter_rows < 50 * k), the plan inverts: a filter-first execution that materialises the filtered slice and then runs lexical + vector scoring across it as a sequential pass, rather than going through the indexes. The EXPLAIN output for that variant looks different:

Hybrid Top-K  k=20  weights=(0.40, 0.60)
  Strategy: filter_first  (filter_rows=147 < 50*k=1000)
  ->  Filtered Materialize  rows=147
        Filter: tenant_id = $1 AND acl_subjects && $2 AND created_at > ...
        ->  Index Scan on chunks_tenant_btree
  Rescored: 147 rows  (lexical + vector, computed on-the-fly)

The lexical and vector scores are computed at scoring time rather than read from the index. That is more CPU per row, but the absolute row count is small, so the trade is correct. It is also the only variant that is guaranteed to find the right answer when the filter is selective — the index-first variants can miss rows entirely when neither the top-100 lexical nor the top-100 vector intersect the filtered slice.

The cost model, briefly

Three estimates drive the strategy choice:

Filter selectivity. Standard stats — histograms, MCVs, distinct-count — same as any planner. Tenant id and time windows are usually well-estimated; ACL overlaps are estimated as a function of the average subject-set size.
Lexical cardinality. Estimated from per-term document frequencies. A query of three rare terms might return 800 rows; a query of three stopwords-after-filtering might return zero, which short-circuits the lexical leg.
Vector recall at a given ef_search. Estimated from a small offline calibration that the indexer runs once per index build and refreshes on compaction. The estimator maps ef_search to expected recall at k for the corpus's distance distribution; the planner inverts it to pick the smallest ef_search that meets the target recall.

The strategy chosen is the one with the lowest expected cost subject to a hard recall floor. The recall floor is the protection against the "looks fast, returns garbage" failure mode: you can configure the floor down, but you cannot configure it away.

What this buys the application

Two things that used to be the application's job are now the planner's.

The application no longer owns the merge. There is one ranked list returned by one statement, with one consistent score, with a degraded annotation when the plan had to fall back. The half-page of TypeScript that used to weight two result sets, deduplicate, and reapply the filter — gone. So is the entire class of bug where the merge code drifted out of agreement with the index configuration.

The application no longer owns the recall/latency trade-off per call site. ef_search, the choice between intersect and union, the choice between index-first and filter-first — these are all decisions the planner makes per query, based on the statistics of the actual data and the actual query. The configuration you ship is a recall floor and a latency budget, not a set of magic numbers tuned for last quarter's corpus.

If you are running a hybrid stack today and have ever had to explain why your top result on a slightly rephrased question is suddenly missing — or why a narrow tenant returns zero hits despite obviously having matching documents — one of the three edge cases above is the explanation. The fix is not a better merge function in application code. The fix is to push the merge into the same plan as the scans, so that the planner can see when a leg has degenerated and react accordingly.

That is the planner work behind hybrid(). The syntax is the boring part.

One WAL, four data models: how cross-modality transactions actually work

2026-05-17

A previous post — The tradeoffs we made to fit four engines in one — said the trick is a single write-ahead log. That sentence is correct, dramatically under-specified, and exactly the kind of thing storage engineers stop trusting after their third bad weekend. So this post is the long version. What the WAL actually contains, how a four-modality transaction lands in it, what the fsync math looks like when a vector batch and a 50 MB blob arrive in the same group commit window, and the four mitigations that keep the cold-blob latency tail from eating the KV path.

If you have never had to reason about a WAL, the short version is: it is the ordered byte stream that every durable change is written to before anything else is allowed to acknowledge the write. Crash recovery replays it. Replication reads it. Backup snapshots are aligned to positions in it. It is the single most load-bearing file in the engine. Putting four data models behind one of these is the design decision that everything else in RedDB falls out of.

What "one WAL" actually means

The temptation, when you have four engines, is to give each one its own log. Postgres has pg_wal, you give the vector engine vec_wal, S3 has its own internal journal, Redis has its AOF. Four logs, four fsync queues, four recovery procedures. It composes — until you want a transaction that spans two of them. Then you are either writing two-phase commit by hand or you are pretending the consistency story is fine and hoping nobody graphs the divergence.

A single WAL means there is exactly one ordered byte stream on disk, and every durable mutation to every modality is appended to it before the client sees an ack. A transaction is, physically, a contiguous run of records in that stream bracketed by a BEGIN and a COMMIT record sharing a transaction id. When the COMMIT record's LSN is durable — meaning fsync returned on the WAL file up to at least that byte offset — the transaction is committed. All of its modality writes are committed atomically because they are all before the COMMIT in the same ordered file. There is no second log to wait on.

This is how a single update can touch a document, append a vector, bump a KV counter and replace a blob and either all four of those changes survive a crash or none of them do. The atomicity is a property of byte ordering on disk, not a protocol.

Record layout

Every WAL record has the same envelope. The body differs per modality, but the envelope does not.

+----------+------+--------+---------+----------+-----------------+--------+
|   LSN    | size |  type  |  txn_id | modality |     payload     |  CRC32 |
+----------+------+--------+---------+----------+-----------------+--------+
| 8 bytes  |  4   |   1    |    8    |    1     | size - 22 bytes |   4    |
+----------+------+--------+---------+----------+-----------------+--------+

LSN — Log Sequence Number, monotonic, assigned at append time. The recovery cursor, the replication cursor, the snapshot cursor are all LSNs.
size — total record bytes including this header. Lets the replayer skip a corrupted record and continue if the CRC fails.
type — BEGIN, COMMIT, ABORT, WRITE, CHECKPOINT, BLOB_REF, FENCE. The first three bracket transactions; WRITE carries actual mutations; CHECKPOINT records a fuzzy checkpoint LSN; BLOB_REF is the sidecar pointer described below; FENCE is a barrier the replay cursor will not cross until acknowledged by the modality engines (used for online schema changes).
txn_id — 64-bit transaction id. Every WRITE carries the id of the transaction it belongs to.
modality — DOC, VEC, KV, BLOB, or META (system records). The replayer dispatches to the right engine using this byte.
payload — modality-specific bytes. The four modalities encode their payload differently:
- DOC — collection id, document id, MVCC version, encoded value (MessagePack).
- VEC — collection id, vector id, dimensionality, raw f32 bytes, optional metadata blob. The HNSW graph mutation is not in the WAL. It is recomputed at replay from the vector id and the existing graph state. This is the single most important encoding decision and we discuss it below.
- KV — namespace, key bytes, value bytes (or a tombstone marker), TTL.
- BLOB — blob id, content hash, size, and a sidecar offset. The blob's actual bytes are not in this record. See the sidecar section.
CRC32 — checksum over the entire envelope including the payload. A failed CRC during replay marks the end of the durable log; anything after is treated as torn.

A transaction touching all four modalities therefore produces, at minimum:

BEGIN(txn=42)
WRITE(txn=42, modality=DOC, ...)
WRITE(txn=42, modality=VEC, ...)
WRITE(txn=42, modality=KV,  ...)
WRITE(txn=42, modality=BLOB, blob_ref=offset_in_sidecar)
COMMIT(txn=42)

Six records, one ordered run, one fsync to make the COMMIT durable. The four modality engines apply their respective WRITE records at replay time, in order, from the same cursor.

The lane diagram

What goes into the WAL is one question. What happens to those records on the way to and from disk is another. Here is the path:

   doc writer       vec writer       KV writer        blob writer
        |                |                |                 |
        v                v                v                 v
  +----------+     +----------+     +----------+     +-------------+
  |  encode  |     |  encode  |     |  encode  |     |  encode +   |
  |  payload |     |  payload |     |  payload |     |  hash blob  |
  +----------+     +----------+     +----------+     +------+------+
        \                \                /                 |
         \                \              /                  |
          \                \            /            +------+------+
           v                v          v             | blob bytes  |
        +---------------------------------+          |  to sidecar |
        |  shared in-memory WAL ring      |          |  (separate  |
        |  buffer (per-CPU sharded,       |          |   fsync     |
        |  global LSN assignment)         |          |   queue)    |
        +-------------------+-------------+          +------+------+
                            |                              |
                            v                              |
                  +---------------------+                  |
                  |   group commit      |  <--- blob ref ---+
                  |   coalescer         |
                  +---------+-----------+
                            |
                            v
                  +---------------------+
                  |   fsync(WAL file)   |
                  +---------+-----------+
                            |
                            v
                     clients ack'd
                  (up to COMMIT LSN)

Three details in that picture matter:

Per-CPU ring shards. The in-memory ring is sharded by core to avoid the lock contention you get when sixteen writer threads all want to append at once. LSN assignment is the only globally-ordered step; payload encoding and CRC happen on the writer's local shard before LSN handoff.
The group commit coalescer. Many transactions' COMMIT records land in the same fsync. A single fsync becomes a single durability event for every transaction whose COMMIT LSN is at or below the flushed byte offset. This is the standard Postgres / InnoDB trick and it is what keeps small-transaction throughput sane on slow disks.
The blob sidecar. Blob bytes never traverse the WAL ring. They go through a separate fsync queue with a different priority. The WAL gets a tiny BLOB_REF record that just points to where the bytes landed in the sidecar. We explain why below; it is the single biggest reason the cold-blob latency tail did not eat us.

Why the vector graph is not in the WAL

If you read the record layout carefully you will notice that a vector WRITE carries the raw f32 bytes but no HNSW graph mutation. That is deliberate and it took us two redesigns to land on.

An HNSW insert produces, in the worst case, mutations to several thousand graph edges across multiple layers. If we logged every edge mutation, the WAL bandwidth from a single 1024-dim vector insert would be 50–100× the bandwidth of the vector itself. Replay would be I/O bound on edge mutations instead of the engine's own batched graph build.

So the WAL stores only the input — the vector and its id — and the graph is treated as derived state. At replay time, the vector engine re-runs the insert against the graph it has in memory. This makes replay slower per vector than a "log everything" approach would be, but it makes the steady-state WAL write rate dominated by the vectors themselves, which is the bound you actually want.

The cost: graph state is not point-in-time recoverable from the WAL alone. It is reconstructable from the WAL plus the most recent vector engine checkpoint. We checkpoint vector engines more aggressively than the other modalities (every five minutes by default, configurable) precisely because the replay cost is higher per record.

The same trick — "log the input, recompute the derived state" — is why we do not log internal LSM compaction events for the document engine, only the logical writes. Document compaction state is rebuilt from checkpoint + WAL replay.

The fsync contention tradeoff

Now the part that bites. Every modality's COMMIT waits on the same fsync. A fsync is not free. On modern NVMe with the write cache flushed correctly (which you must, otherwise your "durable" log is a polite fiction), a single fsync costs roughly 50–200 microseconds in the steady state, climbing to single-digit milliseconds when the device's internal cache is dirty and being drained.

Group commit amortises this for many concurrent transactions, so per-transaction fsync cost is much lower than the raw call cost. But the ceiling on transactions per second is bounded by fsync rate times group size. If your group size collapses — because writers are not arriving fast enough to coalesce, or because the records are huge and the group flush is slow — your tail latency grows.

The pathological case is the one the tradeoffs post named: a burst of blob writes lands in the same window as a steady stream of KV writes, and the KV transactions' COMMIT is stuck waiting for the blob transactions' bytes to flush. That is what produced the 1.8 ms → 14 ms P99 jump we measured under load. The flush window grew because the bytes in the window grew, and small transactions paid the price for being in the same group as large ones.

We are not going to pretend a shared WAL has no contention. It does. The four mitigations below are how we keep the contention from being your problem in the workloads we are appropriate for. Each one is a deliberate cost.

Mitigation 1 — Blob sidecar with pointer-only WAL records

The biggest single win. Blob bytes are not in the WAL. The WAL gets a BLOB_REF record — 64 bytes including the envelope — that points to an offset and length in a separate blob sidecar file. The sidecar has its own fsync queue, its own group commit coalescer, and crucially its own priority relative to the WAL.

The atomicity contract still works because we order the fsyncs: the blob sidecar fsync for a transaction must return before the WAL fsync that carries that transaction's COMMIT. So at recovery time, if you see a COMMIT in the WAL, the bytes it references are guaranteed to be on disk in the sidecar. If the blob fsync succeeded but the WAL COMMIT fsync did not, the transaction is rolled back and the blob bytes are orphaned, to be reclaimed by the sidecar's GC pass. Orphan blobs are cheap to detect (sidecar offsets with no live WAL reference) and reclaim is bounded by checkpoint cadence.

The cost: every blob write costs two fsyncs instead of one. For a workload dominated by tiny KV writes with occasional large blobs, that is a great trade — small writes are no longer queued behind large ones, and the second fsync for the blob is absorbed by a dedicated thread that the KV path never touches. For a workload of nothing but small blobs, it is a worse trade than putting them inline, and we set the inline-vs-sidecar threshold at 32 KB by default for that reason. Below 32 KB, blobs go through the WAL like everyone else.

Mitigation 2 — Group commit with adaptive batching

Group commit is not novel. The tuning is the part that matters.

The coalescer has two thresholds: a time deadline (default 500 µs) and a byte budget (default 1 MB). A flush fires when either threshold is hit. On a cluster with high small-transaction throughput, the byte budget is what bounds latency — you accumulate 1 MB of COMMIT records in well under 500 µs and flush. On a quiet cluster, the time deadline kicks in and a lone transaction waits at most 500 µs before its COMMIT is durable.

What we added on top is a feedback loop on the deadline. If the previous flush took longer than the deadline to complete (because the disk was busy), the next deadline is doubled, up to a ceiling of 4 ms. If flushes are consistently completing well under the deadline, it is halved, down to a floor of 100 µs. This keeps the flush rate matched to what the disk can actually sustain, which prevents a death-spiral where each flush starts before the previous one finishes and the queue grows without bound.

You can disable adaptive batching with wal_group_commit_adaptive = false if you want predictable behaviour for benchmarks. Most operators leave it on.

Mitigation 3 — `wal_priority` hint on the write path

Some writes are happy to commit a little later if it means small writes commit on time. Background ingestion, log-style audit writes, blob commits where the application has already shown the user "uploading…" — none of these need sub-millisecond commit latency.

The write API takes an optional wal_priority hint:

await db.transaction(async (tx) => {
  await tx.docs.put("conversations", convId, conversation)
  await tx.kv.set("counters/messages", count)
  await tx.blobs.put("attachments", attachmentId, blobBytes, {
    wal_priority: "background",  // <-- hint
  })
})

A background priority transaction's COMMIT is allowed to coalesce into the next group commit window instead of the current one. In practice that adds anywhere from 100 µs to 4 ms of latency to the background transaction, and it removes that transaction's bytes from the group that the foreground transactions are queued behind. The application still sees an atomic transaction; the fsync lane is just different.

The hint is opt-in. We do not auto-classify writes as foreground or background — the application knows which user-visible latency budget the write is on, and we are not going to guess.

Mitigation 4 — Per-tenant WAL bandwidth quotas

In a multi-tenant deployment, a single noisy tenant can starve every other tenant by saturating the shared WAL. We measured this once on staging — a single customer's bulk import drove P99 KV latency for every other tenant on the node to 80 ms — and it convinced us to ship fair-share quotas before the next customer hit it in production.

The quota is enforced at the WAL ring entry point. Each tenant has a configurable byte-per-second budget (defaults derived from the plan tier) and a burst allowance. When a tenant exceeds budget, their writes are queued in a per-tenant overflow ring that drains into the shared WAL only when shared bandwidth is available. Other tenants are not blocked by an overflowing tenant's queue depth.

The scheduling is weighted round-robin across tenant rings at the shared-ring entry, not strict priority, so a fully-loaded shared WAL still gives every active tenant proportional throughput rather than starving small tenants entirely. This is the model we mean when we say "fair-share" in the customer docs; the previous round-robin-only scheduler did starve small tenants under load, and we changed it after the same staging incident.

What this gets you

The point of all this machinery is that the application layer can write the obvious code:

await db.transaction(async (tx) => {
  // Document write
  const conv = await tx.docs.put("conversations", convId, {
    title, participants, updated_at: now(),
  })
  // Vector insert (chat embedding)
  await tx.vecs.upsert("conversations.embeddings", convId, embedding)
  // KV counter
  await tx.kv.incr(`counters/messages/${userId}`)
  // Blob commit (attachment)
  await tx.blobs.put("attachments", attachmentId, fileBytes)
})

…and have it be a single transaction. If the process dies between any two of those lines, none of them are visible. If it dies after the COMMIT returns, all of them are. There is no two-phase commit between four engines. There is no compensating write to roll back the vector if the blob upload fails. There is one ordered log, one fsync, one cursor.

This is the thing the stitched stack — Postgres + pgvector + S3 + Redis — cannot give you. You can come close, with careful application-layer patterns and idempotency keys everywhere, and a lot of teams do. But the consistency you get is the consistency you write, not the consistency the storage gives you. The thing we are selling is the inversion of that.

When you should still pick a stitched stack

The shared WAL is the feature and the cost. It is the right architecture for workloads where the seams between modalities are where the bugs live. It is the wrong architecture if the seams are not where your bugs live and your single-modality tuning headroom matters more.

Concretely, do not buy us if:

Your workload is dominated by 100+ MB blob writes at high concurrency and you also need sub-5 ms reads on the same cluster. The blob sidecar mitigates this but does not eliminate it; the disk is still shared.
You need to tune a vector workload to recall-at-99.5% on an unusual embedding model and you know which HNSW knobs you would change. We do not expose them per-collection.
Your isolation requirements are strict serializable and the write-skew workarounds we already documented are not acceptable to your domain.

For everyone else — which in our experience is most teams shipping AI-adjacent applications where the data crosses modalities on every request — the shared WAL is the boring choice. We meant it to be boring. Storage that surprises you is storage you cannot operate.

What lives in this post and what does not

This post is the WAL. Not the rest of the engine. If you want the surrounding context, three companion posts cover the pieces this one assumes:

The tradeoffs we made to fit four engines in one — the bill for sharing a WAL, in language a buyer reads before signing.
Snapshot isolation, in practice — what the consistency model on top of this WAL actually gives an application, and what it does not.
Stop stitching Postgres, pgvector, S3, and Redis — the case for why a shared WAL is worth its bill in the first place.

The deeper specs — the recovery state machine, the replication protocol that reads from the same LSN cursor, the snapshot-and-restore protocol that aligns to checkpoint LSNs — will get their own posts. They all build on the record layout above. If you only remember one diagram from this post, make it the lane diagram. Everything else is implementation.

Postgres + pgvector → RedDB: a migration playbook with wall-clock numbers

2026-05-17

The most common RedDB migration we see is from Postgres with the pgvector extension bolted on. The shape is familiar: a primary OLTP schema, an embeddings table with an HNSW or IVFFlat index, a RAG pipeline that does a vector search joined back to source rows, and an ops team that has started to notice that every schema change to the embeddings table now takes a maintenance window. This post is the playbook we hand to teams making that move.

Numbers below are from a representative test corpus we ran against ourselves before publishing: 12.4M rows in the source table, 1536-dim OpenAI-style embeddings, hot working set ~30 GB. Your numbers will differ — what matters is the ratio between the phases, which is stable across the migrations we have shipped. Treat the absolute milliseconds and minutes as a sanity check, not a contract.

The shape of the source

Almost every Postgres + pgvector workload we have migrated has this skeleton:

CREATE TABLE documents (
  id           bigserial PRIMARY KEY,
  tenant_id    text NOT NULL,
  uri          text NOT NULL,
  body         text NOT NULL,
  metadata     jsonb NOT NULL DEFAULT '{}'::jsonb,
  created_at   timestamptz NOT NULL DEFAULT now()
);

CREATE TABLE chunks (
  id           bigserial PRIMARY KEY,
  document_id  bigint NOT NULL REFERENCES documents(id) ON DELETE CASCADE,
  chunk_idx    int NOT NULL,
  body         text NOT NULL,
  embedding    vector(1536),
  model        text NOT NULL,
  created_at   timestamptz NOT NULL DEFAULT now()
);

CREATE INDEX chunks_embedding_hnsw
  ON chunks USING hnsw (embedding vector_cosine_ops)
  WITH (m = 16, ef_construction = 64);

The hot query is a tenant-scoped top-k followed by a join back to the source document:

SELECT c.id, c.document_id, c.body, d.uri,
       1 - (c.embedding <=> $1) AS score
FROM chunks c
JOIN documents d ON d.id = c.document_id
WHERE c.tenant_id = $2
ORDER BY c.embedding <=> $1
LIMIT 20;

If your workload looks meaningfully different from this — for example, you do not have a tenant column, or you store embeddings in a sibling service — most of what follows still applies, but adapt the dual-write step to your topology.

Phase 0 — pre-flight (1 day)

Before touching either database, write down four numbers. We have not yet seen a migration where these were all known up front, and we have seen migrations stall when they were not.

Embedding cost to rebuild from scratch. Rows × tokens-per-row × $/million-tokens. If this number is more than ~10% of one month of your inference budget, plan to copy embeddings, not regenerate them.
Read QPS, p50/p95/p99 latency. From the actual production load balancer, not from a benchmark. The cutover budget is set by your p99.
Write QPS, including the burstiest hour. Dual-write must survive this without falling behind.
Total bytes on disk for chunks.embedding (SELECT pg_size_pretty(pg_total_relation_size('chunks_embedding_hnsw'))). This sets the wall-clock for the bulk copy.

For our test corpus those numbers were: ~$1,800 to re-embed, 340 read QPS at p95 = 22 ms, write peak 90 QPS, embedding index = 41 GB. We copied embeddings rather than regenerating them. Almost every team should.

Phase 1 — schema in RedDB (1 hour)

The translation is almost mechanical because RedDB speaks the same SQL dialect for table DDL. Two real differences matter:

CREATE TABLE documents (
  id           bigserial PRIMARY KEY,
  tenant_id    text NOT NULL,
  uri          text NOT NULL,
  body         text NOT NULL,
  metadata     jsonb NOT NULL DEFAULT '{}'::jsonb,
  created_at   timestamptz NOT NULL DEFAULT now()
);

CREATE TABLE chunks (
  id           bigserial PRIMARY KEY,
  document_id  bigint NOT NULL REFERENCES documents(id) ON DELETE CASCADE,
  tenant_id    text NOT NULL,          -- denormalised, see below
  chunk_idx    int NOT NULL,
  body         text NOT NULL,
  embedding    vector(1536),
  model        text NOT NULL,
  created_at   timestamptz NOT NULL DEFAULT now()
);

CREATE INDEX chunks_tenant_idx ON chunks (tenant_id);

CREATE INDEX chunks_embedding_ann
  ON chunks USING vector (embedding cosine)
  WITH (lists = 2048, ef_search = 64);

The two differences:

tenant_id is denormalised onto chunks. In pgvector you can filter by tenant via a join, but the planner often picks the vector index first and applies the filter as a post-condition, which is slow when the tenant is selective. Denormalising the column lets RedDB push the predicate down into the ANN search.
The vector index is RedDB's native ANN, not HNSW-via-extension. The WITH parameters look similar (lists, ef_search) but they mean different things — see the RAG-without-a-second-database post for the long-form on tuning. Default to the values above for 1536-dim, ~10M-vector workloads; revisit only if recall is off.

Build the schema, run smoke writes, and confirm the ANN index plan with EXPLAIN. Do not run the migration until you can see the ANN index getting picked.

Phase 2 — dual-write (3–5 days, calendar)

The dual-write window is the lever you pay rent on for the entire migration. Keep it short.

We dual-write at the application layer, not in the database, for two reasons: (a) you keep one transactional boundary per database, which means rollback is just "stop writing to RedDB", and (b) you can shed RedDB writes if it falls behind without disturbing Postgres.

// Pseudocode — adapt to your data access layer.
async function writeChunk(input: ChunkInput): Promise<void> {
  const id = await pg.insertChunkReturningId(input)        // source of truth

  // Best-effort secondary write. Failures go to a retry queue,
  // not back to the user.
  void reddbWriteQueue.enqueue({ id, input })
}

Two non-obvious requirements:

The retry queue must preserve order per document_id. Out-of-order writes will leave you with a chunk in RedDB whose chunk_idx is ahead of a chunk that has not landed yet, and your top-k will return holes. Partition the queue by document_id (or by id, if you never update chunks).
Backfill from Postgres LSN, not from created_at. A created_at-based backfill misses any row that was inserted between your initial SELECT max(created_at) and the dual-write going live. Use pg_export_snapshot() to anchor the bulk copy and start the live writer at the snapshot's LSN.

For the 12M-row test corpus, dual-write ran for four calendar days. Lag stayed under 90 seconds at the worst (the daily reindex run on the source) and under 4 seconds at p95.

Phase 3 — bulk copy with embedding preservation (8–14 hours of wall-clock for ~10M rows)

This is the longest single step. The naïve approach — SELECT * FROM chunks driving INSERT INTO chunks over the wire — will work but takes about 4× longer than necessary and pegs both databases. Do the boring thing instead.

# 1. Snapshot Postgres to a COPY-formatted dump with embeddings as bytea.
psql "$PG_URL" -c "\
  COPY (
    SELECT id, document_id, tenant_id, chunk_idx, body,
           embedding::text AS embedding_text, model, created_at
    FROM chunks
    WHERE created_at <= '${SNAPSHOT_TS}'
  ) TO STDOUT WITH (FORMAT csv, HEADER true)" \
  | zstd -3 -T0 > chunks-${SNAPSHOT_TS}.csv.zst

# 2. Stream into RedDB. Use COPY, not INSERT.
zstd -d -c chunks-${SNAPSHOT_TS}.csv.zst \
  | psql "$RDB_URL" -c "\
    COPY chunks (id, document_id, tenant_id, chunk_idx, body,
                 embedding, model, created_at)
    FROM STDIN WITH (FORMAT csv, HEADER true)"

Three things we wish we had known the first time we did this:

COPY is the only thing that matters for throughput. On the test corpus, COPY-based bulk load ran at ~280k rows/min. The same data via INSERT batched 1000 at a time ran at ~38k rows/min. Same disk, same network.
Build the ANN index after the load, not before. Building during load triples wall-clock and produces a worse index. We use CREATE INDEX post-load, which took 42 minutes on the test corpus.
zstd -3 is the right compression level for a network-bound copy. Higher levels save little additional bandwidth but cost real CPU. Lower levels saturate the disk.

For documents, the same approach — COPY out, compress, COPY in — runs in about a quarter of the chunk time because the rows are smaller and the index footprint is trivial.

Phase 4 — query translation (1–2 days)

Most queries port unchanged. The patterns that need attention:

Vector search with tenant filter

In pgvector the idiomatic version is:

SELECT c.id, c.document_id, c.body,
       1 - (c.embedding <=> $1) AS score
FROM chunks c
WHERE c.tenant_id = $2
ORDER BY c.embedding <=> $1
LIMIT 20;

In RedDB, with tenant_id denormalised onto chunks, the literal translation works and the plan changes:

SELECT id, document_id, body,
       1 - (embedding <=> $1) AS score
FROM chunks
WHERE tenant_id = $2
ORDER BY embedding <=> $1
LIMIT 20;

The join back to documents is usually cheaper as a second round-trip than as a SQL JOIN, because RedDB's planner cannot push the JOIN below the ANN scan and you end up materialising the full top-k before the join. On the test corpus, splitting the query dropped p95 from 31 ms to 19 ms. If your wrapper library makes the round-trip awkward, leave the JOIN in — the absolute difference is small.

Hybrid (vector + lexical) search

If you used pgvector's <=> together with tsvector @@, the translation is almost identical, but in RedDB the planner will pick a parallel scan strategy if you wrap the two scores in a CTE:

WITH
  vec AS (
    SELECT id, 1 - (embedding <=> $1) AS vscore
    FROM chunks
    WHERE tenant_id = $2
    ORDER BY embedding <=> $1
    LIMIT 200
  ),
  lex AS (
    SELECT id, ts_rank(search_v, plainto_tsquery('english', $3)) AS lscore
    FROM chunks
    WHERE tenant_id = $2 AND search_v @@ plainto_tsquery('english', $3)
    LIMIT 200
  )
SELECT c.id, c.body,
       coalesce(v.vscore, 0) * 0.7 + coalesce(l.lscore, 0) * 0.3 AS score
FROM chunks c
LEFT JOIN vec v ON v.id = c.id
LEFT JOIN lex l ON l.id = c.id
WHERE v.id IS NOT NULL OR l.id IS NOT NULL
ORDER BY score DESC
LIMIT 20;

The LIMIT 200 per branch is the lever for recall vs. latency. We start at 200 and tune down only if a recall regression test passes.

`ANALYZE` after the copy

The most common "RedDB is slow" support ticket on a fresh migration is "we forgot to ANALYZE." Run it once after the bulk copy and once after the ANN index build. It takes single-digit minutes on a 12M-row table.

Phase 5 — shadow reads (2–4 days)

Do not flip the read traffic on a Tuesday morning. Shadow-read first.

async function search(query: SearchInput): Promise<SearchResult[]> {
  const primary = await pg.search(query)

  if (sampler.shouldShadow()) {
    void compareInBackground(query, primary)   // never blocks the response
  }

  return primary
}

The shadow compare logs three numbers per request: (a) recall@k between Postgres and RedDB top-k, (b) p99 latency difference, (c) any query that errored on one but not the other. We aim for recall ≥ 0.97 against the Postgres baseline before flipping reads. Our test corpus took 36 hours of shadow traffic to converge there — most of the gap was queries with very small tenants, which is where ANN parameters matter most.

Phase 6 — cutover (15 minutes)

A short, well-instrumented cutover with a one-command rollback. Nothing else.

Set READS_FROM=reddb in the feature flag service (5s).
Watch p95/p99 search latency and error rate for 10 minutes.
If anything looks wrong, set READS_FROM=postgres. You are back where you were because dual-write is still running.
After 24 hours of clean metrics, stop writing to Postgres. Keep it as a hot read replica for one more week.
After the week, decommission.

On the test corpus the cutover itself was uneventful: 12 minutes from flag flip to "we're going to lunch." The interesting work was Phases 2–5; the cutover is just the moment you collect on it.

Rollback plan, explicitly

Rollback is a property of the playbook, not a separate document. The phases above were designed so that at each step you can stop and either back out or hold.

Phase ends	If it goes wrong, you …
Phase 1 (schema)	Drop the RedDB tables. No traffic, no data, no consequence.
Phase 2 (dual-write)	Disable the secondary writer. Postgres is the source of truth and untouched.
Phase 3 (bulk copy)	Truncate the RedDB tables and restart the copy. The snapshot LSN is your idempotency.
Phase 4 (queries)	Keep the old query layer behind the same flag. Roll back at the application boundary.
Phase 5 (shadow read)	Stop shadowing. No user-visible effect.
Phase 6 (cutover)	Flip the read flag back. Dual-write is still running, so no data is lost.

The only step that is hard to reverse is Phase 6 after you have stopped writing to Postgres. That is why "keep writing to Postgres for 24 hours after cutover" is in the playbook with full caps and an exclamation point in our internal version.

The three places teams trip

If you take three things from this post:

Anchor the backfill on an LSN snapshot, not a wall-clock timestamp. Otherwise you will find phantom missing rows weeks later.
Build the ANN index after the bulk load. Doing it during load is the single biggest preventable wall-clock cost of the migration.
Keep dual-write running for 24 hours after the read cutover. Rollback without it is a restore-from-backup; with it, it is a flag flip.

Everything else is mechanical.

What we measured, in one table

Phase	Test-corpus wall-clock	What gates it
0 — pre-flight	~1 day	Stakeholder sign-off on the four numbers
1 — schema	~1 hour	DDL only
2 — dual-write live	4 days calendar	Queue lag stays bounded
3 — bulk copy + index build	11h 24m total	Disk + network
4 — query translation	1.5 days	Engineering effort
5 — shadow read	~36 hours	Recall convergence
6 — cutover	12 minutes	Flag flip + watch

If your corpus is meaningfully bigger than 12M rows or your embeddings are wider than 1536 dims, scale Phase 3 roughly linearly with bytes and budget a longer Phase 5. The other phases are largely independent of corpus size.

If you are in the middle of a migration that does not look like this, send us the shape — there is a good chance we have a variation of the playbook for it.

Threat model: what RedDB protects against (and what it doesn't)

2026-05-17

Most database security marketing reads like the result is the same regardless of the question: encrypted at rest, encrypted in transit, role-based access control, audit logs. Tick, tick, tick, tick. That tells you nothing about which attacks fail and which ones succeed against the system you are about to put production data into.

This post is the opposite shape. It walks the RedDB engine and the vault through STRIDE, names the attacker for each category, and at each step says one of two things: we stop this or we do not stop this — here is what does. The "do not" list is the important half. If your threat model includes one of those attackers, RedDB on its own is not your answer and you should know that before signing the contract, not after.

We have had this conversation with security leads at every regulated-industry buyer we have talked to. The shape of the conversation is always the same: they want to know where the boundary is. So here is the boundary, written down.

The boundary, stated once

RedDB defends the data at rest and the bytes on the wire between replicas. Anything that happens inside a process holding a valid in-memory key — including the RedDB process itself — is outside the boundary. That is not a hedge; it is the design.

Three concrete consequences fall out of that one sentence:

A stolen disk, snapshot, or backup is useless without the master key. Cold-storage breach is the attack we put the most engineering into stopping.
A compromised application process that holds a live decryption key is game over for the rows that key opens. No database-level control fixes this. The fix lives in your process isolation, secrets handling, and BYOK story.
A malicious operator with shell on a running RedDB node can read decrypted hot data. Defence here is operational (least privilege, BYOK with customer-held KMS, audit on the KMS side) — not cryptographic.

Hold those three in your head; the STRIDE walk below is just the long form.

STRIDE, one row at a time

STRIDE is the six-category threat taxonomy from Microsoft: Spoofing, Tampering, Repudiation, Information disclosure, Denial of service, Elevation of privilege. It is not the only frame, but it is the one most security leads have already read, so it is the one we use when they ask.

For each category we name the attacker, what they can do, and whether the engine stops them.

S — Spoofing

Attacker: a network peer impersonating a legitimate RedDB node or client.

What we stop:

Replica-to-replica spoofing. Replicas authenticate to each other with mTLS using a cluster-issued certificate authority. A peer without a cert signed by that CA cannot join the gossip ring, cannot receive replication traffic, and cannot serve reads.
Client-to-cluster spoofing. Client connections are mTLS by default in production profile. The control plane issues short-lived client certs scoped to a tenant; a stolen long-lived API token alone is not sufficient if the operator has enabled the production profile (it is the default; turning it off is a deliberate config change that is logged).

What we do not stop:

A compromised CA. If your private CA is exfiltrated, the attacker can mint valid certs and the engine cannot tell them from a real client. Defence is CA isolation (HSM-backed, ideally) and short cert lifetimes so a leak has a bounded blast radius. This is a property of every mTLS system; we do not pretend to fix it.
In-process impersonation. If a compromised application process has access to your client certificate and key, it is a legitimate client from the engine's perspective. See "I — Information disclosure" below.

T — Tampering

Attacker: someone modifying data at rest, in transit, or in a backup.

What we stop:

Wire tampering between replicas. mTLS provides integrity in transit; a flipped bit on the wire fails the TLS MAC and the connection drops.
At-rest tampering of encrypted rows. Each per-row envelope (g, iv, ciphertext) uses an AEAD (AES-GCM by default). A modified ciphertext fails authentication on decrypt and the read returns an error rather than corrupt plaintext. This holds whether the modification happened in the SSTable file, in a backup, or in a snapshot copy.
At-rest tampering of unencrypted columns. SSTables carry block-level checksums (xxh3) that are verified on read. A bit-flip in a non-vault column surfaces as a read error, not silently corrupt output.

What we do not stop:

A privileged operator with write access to the disk path. Truncating an SSTable, deleting a WAL segment, or rolling back a snapshot is indistinguishable from legitimate ops at the cryptographic layer. Defence is file-system permissions and an immutable backup destination (object-lock S3, or equivalent). The engine assumes its own disk is trusted.
A compromised application that issues legitimate writes. If the attacker can call PUT, the writes are authentic from the engine's perspective. This is intentional — the engine is not in the business of policing application-level semantics.

R — Repudiation

Attacker: someone who wants to deny they did a thing.

What we stop:

Write attribution. Every mutation carries the client identity (subject DN from the mTLS cert) into a transactional audit log. The audit log is written in the same transaction as the data and is one of the streams the hooks layer emits; if the write commits, the audit row commits.
Audit log tampering. Audit rows are vault-encrypted with a generation-pinned key separate from the application key, so an operator who can rotate the application key cannot quietly rewrite history.

What we do not stop:

An attacker who has the legitimate client cert. The audit log will faithfully attribute their writes to whoever owns that cert. Repudiation defence at this layer is identity hygiene, not cryptography.
Backdated entries written by a compromised RedDB process. The audit log is not externally notarised. If you need stronger non-repudiation (regulator-grade), stream the audit log to an append-only external system (S3 with object lock, a managed audit service) and treat the engine's copy as a cache. We have customers doing this; it is wired through the same hooks API.

I — Information disclosure

This is the category most buyers care about and the one with the most nuance. We split it by where the data lives.

I.1 — Data at rest, on RedDB's own disks

Attacker: someone who walks off with the disk, the snapshot, or the backup.

What we stop:

Every row in a vault-protected column is AES-GCM encrypted with a DEK wrapped by the current master key generation. The disk on its own is ciphertext.
WAL segments containing those rows are encrypted with the same envelope before fsync. A leaked WAL does not leak plaintext.
Backups are written through the same vault path. Restoring a backup requires access to the master key generations that were active when the backup was taken; the backup file on its own is useless.

What we do not stop:

Unencrypted columns. The vault is opt-in per column. If a column is not in the vault config, its plaintext is on disk. This is a deliberate trade — full-database encryption defeats most engine-level features (sort, hybrid search, filters that read the column). Pick what to encrypt; we will not pick for you.
An attacker who steals the disk and the live master key. If your KMS credential file is on the same machine as your data files, theft of the machine is theft of both. See BYOK and external KMS.

I.2 — Data in transit

Attacker: a network observer between replicas or between client and cluster.

What we stop:

mTLS, modern cipher suites, no TLS 1.0/1.1, no static-key Diffie-Hellman. Wire is opaque.
Replication includes ciphertext from the vault path, not plaintext re-encrypted at the link. A passive network observer on the replication link sees the same opaque bytes the disk holds.

What we do not stop:

A man-in-the-middle who has a valid cert. Same caveat as Spoofing.
Side-channel disclosure via traffic shape. An observer who sees the size and timing of replication packets can learn something about write rate per tenant. This is true of every replicated database and is not part of our threat model.

I.3 — Data in memory, in the RedDB process

Attacker: someone with shell on a running RedDB node.

What we do not stop, and we want this called out plainly:

Hot rows in the RedDB process memory are plaintext after decrypt. A root-shelled attacker can gcore the process and recover plaintext for any data the process has recently read.
Master keys are in process memory while the process is running. We zero them on shutdown and on rotation, but a live process holds live keys.

Defences (operational, not cryptographic):

Least privilege: nobody has interactive shell on production RedDB nodes; access is brokered, audited, time-boxed.
BYOK with a customer-held KMS turns this from "the vendor can read your data if compromised" into "the vendor can read your data only while a session against your KMS is active". The customer can pull the KMS grant and the next key-cache eviction renders the engine unable to decrypt new generations.
HSM-backed KMS with remote-unwrap (HYOK) reduces this further; we mention it but it is not the default.

I.4 — Data in memory, in the application process

Attacker: anyone who can compromise your application.

What we stop: nothing. This is outside the boundary. The vault gives you plaintext when you ask for it through a legitimate client. A compromised application is a legitimate client.

Defences: yours. Process isolation, secrets management, request-scoped decryption (only decrypt the row the current request needs), and BYOK so a panic-revoke is possible.

D — Denial of service

Attacker: someone trying to make the database stop serving traffic.

What we stop or mitigate:

Per-tenant resource quotas. A noisy tenant cannot eat the whole node's IO or memory budget; the scheduler enforces a fair-share with hard ceilings.
Connection-level rate limits at the gateway in front of the engine.
Backpressure under write pressure rather than unbounded queue growth; clients see slow-downs before the engine OOMs.

What we do not stop:

Volumetric attacks below the gateway. A real DDoS is a perimeter problem; the engine is not your edge.
An expensive-but-valid query pattern. Hybrid search with a deliberately bad filter selectivity will burn CPU. We surface that in query observability so you can detect and rate-limit it at the application layer, but the engine will execute what it is told.
Disk-fill from a privileged tenant. Quotas protect against tenant-vs-tenant, not against the cluster operator pushing the disk to 100%.

E — Elevation of privilege

Attacker: a low-privileged client trying to act as a higher-privileged one.

What we stop:

Tenant boundary. Every read and write carries a tenant identity derived from the mTLS subject; the planner refuses to return rows outside that tenant unless the caller has an explicit cross-tenant grant. This is enforced at the storage layer, not just in the API.
Row-level access checks. Where row-level grants are configured, they are evaluated after the vault decrypt but before the row leaves the process, so a query that touches forbidden rows still cannot exfiltrate them.

What we do not stop:

Misconfigured grants. If you grant * to a role, the engine will honour that grant. We log it loudly at config time but do not refuse it.
A compromised admin credential. Admin is admin. There is no cryptographic guard that a determined admin cannot turn off. The defence is admin hygiene plus an audit trail that an external observer can read after the fact.

What this means for buyers

If the regulator asks "is the data encrypted at rest" the answer is yes. That is not the interesting question. The interesting questions are below, with the answers:

Question	RedDB's answer
Can a stolen backup be opened without the master key?	No.
Can a network observer between replicas read row contents?	No.
Can a compromised RedDB process read hot rows?	Yes, while it holds the keys.
Can a compromised application process read its own tenant's rows?	Yes, by definition — it is a legitimate client.
Can RedDB the vendor read your data?	Only with valid KMS access. In BYOK mode, only while you grant it; pull the grant and we go dark on the next cache eviction.
Can an admin with shell rewrite audit history?	Not silently — the audit log is vault-encrypted with a separate key generation, and you should mirror it to an external append-only store if you need regulator-grade non-repudiation.
Can the engine prevent a malicious operator on your side?	No. That is your problem; we make it auditable.

If any "yes" in that table is unacceptable for your threat model, the mitigation is named in the relevant STRIDE section above. If a mitigation we name is still insufficient — for example, you need HYOK with remote-unwrap on every read — talk to us; that is a deployment shape we have built before but it is not the default.

What we are not claiming

To close the loop, an explicit list of things this engine does not do, that other vendors sometimes imply they do:

Homomorphic computation on encrypted data. We do not. Vault columns are decrypted in-process to be operated on.
Confidential-computing-style memory enclaves (SGX/SEV). Not a default. We have customers running RedDB inside an enclave for the I.3 attacker, but it is a deployment choice they make and operate; the engine does not require it.
Provable deletion across replicas. GDPR-style erasure works through crypto-shredding — destroy the key, the ciphertext becomes noise. This is the right answer for replicated systems but it is not the same as overwriting every copy of the plaintext bit-for-bit.
Defence against a state-level adversary with physical access. That is not a database problem.

Naming these gaps is the point of this post. A buyer who reads the security one-pager and does not see them listed should ask why.

This page is a snapshot. The cryptography, the audit-log shape, and the BYOK story are all moving targets; the boundary is not. If anything in the STRIDE walk changes, we update this post — it is the canonical reference we point security reviewers at, so it is in our interest to keep it honest.

Reviewed by RedDB's security lead before publication. If you want the deeper write-up — full STRIDE matrix, deployment-shape variants, sample audit-stream consumer — it is on the docs site under security/threat-model.

RedDB as Claude Code's memory backend — beyond CLAUDE.md

2026-05-16

The problem with CLAUDE.md

Every CLI agent ends up reinventing the same hack: a markdown file at the project root that gets shoved into the system prompt on every turn. Claude Code calls it CLAUDE.md (plus a per-project MEMORY.md). Cursor has .cursorrules. Codex has AGENTS.md. The shape is the same and the failure modes are the same:

It bloats context. Every turn re-reads the whole file. By month three the file is 12 KB of half-stale rules nobody trims.
It is not queryable. "What did the user say about retries last month?" requires grep, not a query.
It is single-scope. One file per project. Cross-project memory (your shell preferences, your code-review tone) has to be duplicated or symlinked.
It is not auditable. Who wrote which line? When? Why? git blame if you remembered to commit it.

A database fixes all four. The interesting question is how to wire it in without forking the CLI.

The shape of the fix

Three pieces:

A schema for memory rows — kind, scope, body, embedding, timestamp.
A SessionStart hook that runs once when the agent boots and injects the top-K relevant memories as context.
A /remember slash command that writes a memory row from inside a session.

All three talk to the same RedDB instance. The transactional boundary means a write from /remember is durable before the slash command returns, and the next SessionStart will see it.

Schema

A single table — call it memory — with five columns plus a vector index.

CREATE TABLE memory (
  id          TEXT PRIMARY KEY,           -- ulid
  kind        TEXT NOT NULL,              -- user | feedback | project | reference
  scope       TEXT NOT NULL,              -- global | <project-path> | <session-id>
  body        TEXT NOT NULL,              -- markdown, what the model reads
  embedding   VECTOR(1024) NOT NULL,      -- cohere-embed-v4 or similar
  created_at  TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE INDEX memory_scope_idx ON memory (scope, created_at DESC);
CREATE INDEX memory_embedding_idx ON memory USING hnsw (embedding vector_cosine_ops);

kind mirrors the categories Anthropic's own memory skill uses (user / feedback / project / reference). scope lets a single instance serve global preferences alongside per-project facts. The HNSW index is what makes "top-K relevant" cheap.

The SessionStart hook

Claude Code fires a SessionStart event when a session boots. A hook is any shell command that returns JSON on stdout — Claude reads it and merges the result into the conversation as a system-reminder.

Drop this in ~/.claude/hooks/session-start-memory.sh:

#!/usr/bin/env bash
set -euo pipefail

SCOPE_GLOBAL="global"
SCOPE_PROJECT="$(pwd)"
TOP_K="${REDDB_MEMORY_TOP_K:-12}"

# CWD + last user message gives us a query embedding seed.
# At SessionStart there is no user message yet, so we seed with
# the project path. /remember will re-rank against real queries later.
QUERY="agent session start in ${SCOPE_PROJECT}"

curl -sS \
  --fail \
  --max-time 2 \
  -H "Authorization: Bearer ${REDDB_TOKEN}" \
  -H "Content-Type: application/json" \
  "${REDDB_URL}/memory/recall" \
  -d "$(jq -nc \
        --arg q "$QUERY" \
        --arg g "$SCOPE_GLOBAL" \
        --arg p "$SCOPE_PROJECT" \
        --argjson k "$TOP_K" \
        '{query:$q, scopes:[$g,$p], top_k:$k}')" \
  | jq '{
      hookSpecificOutput: {
        hookEventName: "SessionStart",
        additionalContext: ( .rows
          | map("- [\(.kind)] \(.body)")
          | join("\n")
          | "## Recalled memory\n\n\(.)"
        )
      }
    }'

{
  "hooks": {
    "SessionStart": [
      {
        "matcher": "startup",
        "hooks": [
          { "type": "command", "command": "~/.claude/hooks/session-start-memory.sh" }
        ]
      }
    ]
  }
}

Two details worth flagging:

--max-time 2. If RedDB is unreachable the hook fails fast and the session boots without memory rather than hanging.
The output uses Claude Code's hookSpecificOutput.additionalContext shape, which is appended as a system-reminder. This is the only way to inject text into the context window from a hook without invoking the model.

The `/remember` slash command

Slash commands in Claude Code are markdown files under ~/.claude/commands/. The frontmatter declares argument hints; the body is a prompt for the model. Here we use the ! prefix to run a shell command inline.

~/.claude/commands/remember.md:

---
description: Persist a memory to RedDB
argument-hint: <kind> <body...>
---

Persist this as a `$1` memory in RedDB, scoped to the current project.

!curl -sS \
  --fail \
  -H "Authorization: Bearer $REDDB_TOKEN" \
  -H "Content-Type: application/json" \
  "$REDDB_URL/memory/write" \
  -d "$(jq -nc \
        --arg kind "$1" \
        --arg scope "$(pwd)" \
        --arg body "${@:2}" \
        '{kind:$kind, scope:$scope, body:$body}')"

Confirm in one line. Do not summarise the body back.

Usage from inside a session:

/remember feedback Don't mock the database in integration tests — we got
burned last quarter when mocked tests passed and the prod migration failed.

The shell call writes the row; RedDB computes the embedding server-side (cheaper than embedding in the hook), commits, and returns the row id. The model gets the JSON response in its tool result and acknowledges.

The retrieval query

The /memory/recall endpoint is a thin wrapper over a single SQL query. It is worth showing because the shape of the query is the whole point of using a database:

WITH q AS (
  SELECT embed($1) AS v
)
SELECT id, kind, scope, body, created_at,
       1 - (embedding <=> q.v) AS score
FROM memory, q
WHERE scope = ANY($2::text[])
ORDER BY embedding <=> q.v
LIMIT $3;

One statement does scope filter + vector ranking + score. Compare to the markdown-file world, where filtering across global + project scope means concatenating two files in the right order and hoping the model picks the relevant lines out.

Why this matters

Three things you get for free that a flat file cannot give:

Scope composition. Global preferences + per-project facts + per-session scratch, ranked together, deduplicated by id.
Decay. A created_at column plus a half-life term in the ranking (score - 0.01 * age_in_days) lets stale memories fall off the top-K without anyone curating the file.
Audit. Every memory has an id and a timestamp. A second table memory_audit (id, action, actor, ts) gives you a git blame for facts the agent acted on.

None of this is exotic. It is the database treating agent context as just another query workload. The CLI changes are 80 lines of bash and JSON.

What is next in this pillar

This post is the anchor. The follow-ups make the pieces concrete:

A walkthrough of building a remember-this skill end-to-end in 30 minutes.
Slash commands with memory — /remember, /forget, /recall — and the queries underneath.
Hooks that mutate persistent state, made transactional so a half-finished hook doesn't leave you with a phantom row.
An MCP server that exposes the same memory.* surface to every MCP-compatible CLI: one DB, all the agent tools.

If you want to be told when those land, grab the Atom feed.

Hello from inside the engine

2026-05-16

Why another database

Most teams glue together three or four engines to ship a single product: a row store for OLTP, a search index for retrieval, a vector store for RAG, an object store for blobs. Each one has its own consistency model, its own failure surface, its own ops playbook.

RedDB collapses these into one engine with one transactional boundary. You write once, you query across modalities, you don't reconcile.

What this blog will cover

Internals of the storage layer — LSM trees, segment compaction, snapshot isolation.
RAG primitives that live next to your transactional data.
The vault: per-row encryption with key rotation that doesn't require downtime.
Honest notes on what we got wrong before 1.0.

This is the first post. Grab the Atom feed and we'll show up in your reader as new pieces land.

Stop stitching Postgres, pgvector, S3, and Redis

2026-05-16

The default stack for a serious application in 2026 looks like this: Postgres for rows, pgvector or a sidecar like Qdrant for embeddings, S3 (or R2, or MinIO) for blobs, Redis for the hot path. Four engines. Four query languages, if you count "S3 GET" as a query language. Four sets of credentials, four backup strategies, four ways the system can be down on a Saturday night.

You did not pick this stack. The stack picked you. Each engine arrived for a defensible reason: Postgres because it is the safe boring choice and you needed transactions; pgvector because the embeddings had to go somewhere and the Postgres extension was easier than spinning up another service; S3 because the PDFs and audio clips would not fit in a row; Redis because somebody put SELECT inside a tight loop and the read replicas could not keep up.

Each decision was correct in isolation. The aggregate is what is killing your team.

The four bills you are paying

When people compare databases, they compare benchmarks. Throughput, P99, cost per gigabyte. The benchmarks miss the actual line item, which is paid in engineer-hours and pager fatigue, not microseconds.

Pull up the last twelve months of incidents on any team running the four-engine stack and you will find the same pattern: most of the outages live in the seams. The database is fine. The cache is fine. The blob store is fine. What broke is the assumption that all three agreed about the state of the world at the same moment.

Here is the bill, in four lines.

Bill #1: Four failure surfaces. Postgres has its own failover semantics. Redis has its own (and it is not the same one). S3 has eventual consistency on some operations and strong consistency on others depending on the year and the vendor. pgvector inherits Postgres but its index build is its own beast. Each of these has its own way of paging you at 3am, its own runbook, its own "yes we have seen this before" tribal knowledge. The number of distinct ways the system can be in a bad state is roughly the product, not the sum.

Bill #2: Four consistency models. Postgres gives you snapshot isolation by default. Redis gives you "whatever was in memory until the last fsync, and trust us about the fsync." S3 gives you read-after-write for new objects and eventual consistency for overwrites (and there is a footnote per region). pgvector gives you Postgres semantics for the rows and an HNSW graph that does not participate in your transaction at all. Stitching these together means writing application code that pretends one consistency model exists. That code is wrong. You just have not hit the bug yet.

Bill #3: Four ops playbooks. Postgres needs pg_upgrade rehearsals, replica lag dashboards, autovacuum tuning. Redis needs eviction policy choices, persistence tradeoffs, cluster reshard procedures. S3 needs lifecycle rules, multipart upload retries, bucket policy audits. pgvector needs index rebuild plans when you change the embedding model. Each is a binder of operational knowledge. New hires read four binders before they are useful on call. Senior engineers maintain four binders or, more honestly, two of them and a wiki page that says "ask Maria."

Bill #4: Four authentication and authorization surfaces. Postgres roles, Redis ACLs, S3 IAM policies, and whatever you bolted on for the vector store. Each system has its own concept of "user," "permission," and "audit log." Centralizing this is its own project. Most teams do not centralize it; they accept the sprawl and hope the next compliance audit is lenient.

None of these bills show up in the database vendor's pricing page.

The integration tax

The four engines do not talk to each other. Your application is the integration layer. Every interesting business operation is therefore a distributed transaction, and most teams handle distributed transactions by pretending they do not have them.

Consider the simplest possible AI feature: a user uploads a PDF, the system extracts text, generates embeddings, and makes the document searchable. On the four-engine stack:

Write metadata row to Postgres.
Upload PDF to S3.
Push extraction job to Redis queue.
Worker pulls job, fetches PDF from S3, computes embeddings.
Worker writes embeddings to pgvector.
Worker updates Postgres row with status = 'indexed'.

How many of those steps can fail independently? Six. How many of those failures leave the system in a consistent state on their own? Zero. Every team I have talked to has at some point shipped a version of this with a bug where the row says indexed but the embeddings are missing, or the embeddings exist but the row was rolled back, or the S3 object was uploaded but the row never got written and the orphan sits in a bucket until somebody runs a reconciliation script that turns out to also have a bug.

You can fix this. The fixes are real engineering work. You write idempotent workers. You design compensating transactions. You build a reconciler. You write a runbook for "what to do when the embeddings job dies between steps 4 and 5." You staff a team to maintain the reconciler.

Or you write one transaction.

"But Postgres can do all of this"

Yes, sort of. This is the response I expect from anyone who has shipped on Postgres + pgvector + lo (large objects) and got it working. The shape of the response is: "we run one engine, we did not have to integrate anything, your manifesto is invalid."

The honest version of that argument: Postgres handles the document and vector path well, lo is fine for moderate blobs, and a queue can live in a table. This is true at small scale and for some workloads. We wrote a whole companion post titled When you don't need RedDB saying so.

Here is where the argument breaks. The moment your blobs are large enough that you do not want them in the WAL, you move them to S3. The moment your embedding workload outgrows pgvector's index build window, you spin up Qdrant or Weaviate next to it. The moment your queue throughput needs durability semantics that a Postgres table cannot give you cheaply, you reach for Redis Streams or Kafka. The single-engine stack is stable until the workload changes, and then it pulls itself apart engine by engine and you are back to four.

The four-engine stack is not a choice. It is the stable equilibrium that Postgres-only teams drift into.

What "small team unlock" means

The common framing of multi-model databases is "scale." You hear "consolidated storage" and assume the pitch is about saving infrastructure cost at high scale.

That framing is wrong, or at least incomplete. The bigger benefit is at small scale.

A team of three engineers cannot run four engines in production responsibly. They can run them; the question is whether they can run them well. A four-engine stack assumes you have someone who knows Postgres replication, someone who has been paged for a Redis OOM at least three times, someone who has fought S3 IAM, and someone who has rebuilt a pgvector index after switching embedding models. At three engineers, this person is you, and you have to be all four people in a single brain. That is the actual cost.

At fifteen engineers, you have specialists, but now the specialists have created silos. The team that owns the vector store does not own the source-of-truth row in Postgres. When the embeddings get stale, ownership of the bug is unclear. The drift window — the time between source data changing and embeddings being recomputed — becomes a customer-visible bug surface and a political question about which team's queue takes priority.

A multi-model engine collapses that. One transaction. One ownership boundary. The reconciler does not exist because the inconsistency it reconciles does not exist.

We will go deep on the ops cost math in a follow-up: Why best-of-breed loses at small scale. The short version is that engineer-hours dominate infrastructure cost at every team size you are likely to operate at, and engineer-hours scale with the number of distinct systems you have to keep alive.

The performance argument is the weakest one

Best-of-breed proponents will note that a purpose-built vector store will beat a multi-model engine on raw vector search benchmarks. They are right. We do not pretend otherwise — read our own tradeoffs post for what we gave up to fit four engines under one transaction.

But this is the wrong axis to fight on. The performance argument is the weakest argument for the four-engine stack, because the performance gap closes every quarter and the operational gap does not. HNSW implementations are converging. JSON indexing in row stores is good enough for most document workloads. The blob path is the blob path. What does not converge is the cost of running four systems versus one. That cost is structural. It comes from the integration layer being your application code, which is to say, code your team writes and maintains.

If your vector search workload genuinely needs the last 15% of throughput that a single-purpose store provides, run a single-purpose store. We will tell you to. But "we benchmark 15% slower on synthetic vector recall" is a strange reason to accept four times the on-call burden.

What we are actually arguing

The argument is not "RedDB is faster than your four engines." Sometimes it is, often it is not. The argument is that the cost of stitching has been quietly moved off the database vendor's bill and onto your team's calendar, and that move is not free.

You are paying for the integration layer in code your team writes, in incidents that pull people away from features, in onboarding ramps that take twice as long because there are twice as many systems to learn, in audit findings that say "your access control surface is not consistent across data stores."

A multi-model engine is the bet that those costs are higher than the cost of giving up some per-modality tuning. For the teams we have worked with — and especially for the teams of two-to-fifteen who are shipping product, not running infrastructure — that bet pays off. For teams running specialized workloads at massive scale with a dedicated platform org, it may not. Honesty about which side of the line you are on is the whole point.

If you are on the four-engine treadmill and the last incident you handled lived in the seams between two of them, the cost of stitching is not theoretical for you. It is the meeting you canceled this week because the reconciler was paging again.

We built RedDB because we wanted to stop having that meeting.

Where to go from here

This is the anchor of a series. Over the next few weeks we will go deeper on each face of the argument:

The tradeoffs we made to fit four engines in one — what we gave up. Honest.
When you don't need RedDB — the cases where Postgres alone is the right call.
Why best-of-breed loses at small scale — the engineer-hours math.

If you want to see what a single transaction across document, vector, KV, and blob actually looks like, the RAG without a second database post has the schema and the code.

The first step is to count, on your current stack, how many of your last ten incidents lived in the seams. If the answer is more than two, you already know what this post is about.

The tradeoffs we made to fit four engines in one

2026-05-14

Picking a storage model is a constraint on every future feature. We picked four. A document store, a vector index, a KV cache and a blob keeper — all riding the same write-ahead log, all visible inside the same transaction. The pitch writes itself. The bill is harder.

This post is the bill. If you are evaluating RedDB against a stitched stack (Postgres + pgvector + S3 + Redis) or against a single best-of-breed engine, you should leave knowing exactly which of those alternatives still beats us on your workload. We would rather lose the deal than win it and own the incident.

One log to rule them all

A single WAL is shared by every modality. Document writes, vector inserts, KV updates and blob commits land in the same ordered byte stream and are made durable by the same fsync. That is the whole trick. Cross-model transactions are not a coordination protocol bolted on top of four engines — they are the natural consequence of one engine that already happens to write all four things.

Everything else in this post is a downstream consequence of that one decision. Some of the consequences are good (you only need one backup, one auth surface, one ordered cursor for change capture). Some of them are not. We will start with the not.

Tradeoff 1 — Per-modality tuning surface

A pure vector store like Qdrant or Weaviate exposes every HNSW parameter that matters: graph degree M, build-time candidate list ef_construction, query-time candidate list ef_search, the choice between cosine, dot and L2, scalar vs product quantization, segment-level rebuild scheduling. You can tune one collection for recall-at-99% on a 768-dim text embedding model and a different collection for sub-millisecond response on a 64-dim recsys model, and they coexist happily.

We did not ship that. The vector index in RedDB exposes three knobs:

index_type — HNSW or flat. Flat is for collections small enough that brute force wins.
distance — cosine, dot or L2.
ef_search — the query-time tradeoff between recall and latency, set per query.

We picked sensible defaults for M (16) and ef_construction (200). They are not configurable per collection. If your workload genuinely benefits from M = 48 (very high-dimensional embeddings where recall collapses at low graph degree), we are leaving recall on the table for you. We made this call on purpose: the support load of explaining HNSW parameters to teams that picked us because they did not want to learn HNSW parameters was the load we wanted to avoid. A team that wants those knobs is a team for whom a dedicated vector store is the right answer.

The same logic applies elsewhere. Blob storage does not let you pick the chunk size. The KV layer does not let you opt out of MVCC for raw byte speed. The document store does not let you choose between B-tree and LSM per collection. These are all real knobs in real specialized engines. We cut them.

Tradeoff 2 — The plugin story we did not ship

Postgres has spoiled the industry. The extension API is the reason pgvector exists at all, and it is the reason a small team can ship a new index type without forking the database. Every serious Postgres engineer has, at some point, written or read a CREATE EXTENSION shim.

We do not have that. RedDB is one binary with the four engines compiled in. You cannot drop in a third-party graph index, a new compression codec, or a domain-specific aggregate function without forking us. We considered building a plugin ABI; we decided not to. Two reasons:

A plugin ABI is a forever commitment. The Postgres ABI is famously stable and famously load-bearing on the entire ecosystem. We are a young engine. Locking ourselves into a plugin contract today means freezing internal APIs we still want to rearrange.
Most plugin demand is really feature demand. When we asked teams what they would build as a plugin, the answers were almost always "a feature I wish you shipped." Building those features in tree, where they participate in the WAL and the transaction model, is strictly better than pushing them outside the trust boundary.

If you are the team that would have written the plugin — your differentiation lives in a custom index, a custom aggregate, a custom rerank — you should keep using a database that lets you build it. We will not be that database in 2026. We might be in 2028. Plan accordingly.

Tradeoff 3 — The cold-blob latency tail

Here is the one that bites in production. The WAL is shared. That means a 50 MB blob commit and a 200-byte KV update are queued behind the same fsync. On a healthy node with NVMe and a tight wal_segment_size, this is fine — the blob is striped, the small writes pipeline, and tail latency stays sane. On a node that has just absorbed a burst of large blob writes, it is not fine.

The numbers we measure on our own staging cluster:

KV P99 under steady load: 1.8 ms.
KV P99 during a sustained blob write burst (10 concurrent uploads, 100 MB each): 14 ms.
KV P99.9 during the same burst: 38 ms.

For most applications, a 14 ms P99 during ingestion bursts is invisible. For a hot path doing 50 KV reads per request, it is a 700 ms request tail you did not expect. We document this. We expose a wal_priority hint that lets you mark blob commits as background, which softens the tail at the cost of slightly delayed blob durability acknowledgement (still durable, just acknowledged after the next group commit instead of with it).

If your workload is "a few hundred KV ops per second and never more than a megabyte of blob per minute," none of this matters and you should pick us. If your workload is "ingest 4 TB of PDFs a day while serving sub-5ms cache reads on the same cluster," put the blobs in S3, put the cache in Redis, and use us for the documents and vectors. We are not lying about the shared WAL. The shared WAL is the feature. The shared WAL is also the cost.

Tradeoff 4 — Operational maturity

We are honest about being younger than the alternatives we replace. Postgres has thirty years of operational folklore. S3 has fifteen years of corner-case documentation. Redis has the entire redis-cli ecosystem and a generation of engineers who have debugged it at 3am.

We have:

A two-year-old engine.
A small team.
A growing but not encyclopedic set of postmortems.
Documented behavior for the failure modes we have seen; less coverage for the ones we have not.

This is not a lasting tradeoff — the maturity gap closes one quarter at a time — but it is a real one today. If your team has zero appetite for being on the early side of an operational learning curve, the four-engine stack you already know is the safer choice. Buying RedDB to escape it is rational only if the seams in that stack are already costing you more than the maturity gap will.

When RedDB is wrong for you

To make this concrete, here is the shortlist. If you recognize yourself in any of these, do not buy us yet:

You need every HNSW knob. A dedicated vector DB will give you better recall-at-latency on a single embedding workload than we will.
Your differentiation is a custom index, aggregate or codec. Postgres + an extension is a better home until we ship a plugin ABI.
You ingest huge blobs while serving low-latency reads on the same cluster. Split the workload. Use object storage for blobs.
You have a strong on-call team that already loves the four-engine stack. The seams you have memorized cost less than a migration.
You cannot tolerate a young engine. Wait a year. We will still be here.

What we kept

What we did keep is exactly the thing the bill in the previous post was buying out: cross-model transactions, one ops playbook, one auth surface, one ordered log of every change to every modality. For the teams whose pain lives in the seams rather than in single-engine tuning, that trade was obvious. For the others, the four-engine stack is still the right answer and we will say so on a call.

Honest tradeoffs are the only kind that survive contact with production. These are ours.

RAG without a second database

2026-05-12

Most RAG stacks treat the vector store as a sibling of the source-of-truth database. The text lives in Postgres or Mongo, the embeddings live in Pinecone or Qdrant or a pgvector table, and a job somewhere is responsible for keeping them in agreement. That job is where the bugs come from.

This post is about the specific class of failure that disappears when the embedding column lives on the row it describes — and about the schema, transactions, and queries that make that possible.

The two-store assumption nobody questions

Open any RAG tutorial published in the last two years. The architecture diagram has, with depressing consistency, three boxes: a document store, an embedding model, and a vector database. An arrow goes from the document store to the model. Another arrow goes from the model to the vector store. At query time, the application embeds the user's question and asks the vector store for nearest neighbors.

The diagram is clean. The implementation is not. The arrows hide a small distributed system.

In production, the arrow from "document store" to "embedding model" is a queue. A worker reads new and changed rows from the source database — sometimes by polling a updated_at column, sometimes by tailing a CDC stream, sometimes by a trigger that writes to an outbox table. The worker calls the embedding model, then writes the resulting vector to the vector store, keyed by the source row's primary key.

If anything in that pipeline lags, fails, retries, or is deployed mid-flight, the vector store and the document store now disagree about the world. A query lands in this window. The user sees the bug.

The drift window, named

Call this the drift window: the interval, measured per row, between the moment a document changes and the moment its embedding in the vector store reflects that change.

For a healthy pipeline running well, the drift window is short — seconds, sometimes minutes. For a pipeline under load, after a queue backlog, during a re-embed migration, or right after a deploy, the drift window is hours or worse. We have seen production drift windows in the tens of hours, undetected, because there was no alarm watching the gap.

What does the user experience inside the drift window?

A support article was updated this morning to reflect a price change. The embedding still encodes the old paragraph. The user asks "how much does X cost?" Retrieval pulls the article (correctly), but the snippet the model is grounded on — the chunk the retriever scored highest — still semantically matches the old wording. The answer is wrong.
A legal team redacted a name from a document. The text on disk is clean. The embedding was generated from the unredacted text. Vector similarity search on the old name still pulls that document into context. The model dutifully includes the redacted name in its answer.
A document was deleted from the canonical store. The vector remained, because the deletion event was on a different topic and the consumer for that topic was down. Search returns a ghost — a result whose source no longer exists. Click-through returns 404 or, worse, returns the previous tenant's row.

Each of these is a real outage we have seen on customer postmortems. Each is mechanical, not stochastic. Each disappears when there is no second store to drift against.

Schema: the embedding belongs on the row

The mental shift is small but consequential. Stop thinking of "the embedding for document 17" as a separate object that has to be kept in sync. Start thinking of it as one more column on document 17.

Here is the table:

CREATE TABLE documents (
  id          UUID PRIMARY KEY,
  tenant_id   UUID NOT NULL,
  title       TEXT NOT NULL,
  body        TEXT NOT NULL,
  updated_at  TIMESTAMPTZ NOT NULL DEFAULT now(),

  -- the embedding lives here, on the same row, in the same engine
  body_embedding  VECTOR(1024) NOT NULL,
  embedding_model TEXT NOT NULL,         -- which model produced it
  embedded_at     TIMESTAMPTZ NOT NULL   -- and when
);

CREATE INDEX documents_tenant_embedding_idx
  ON documents
  USING hnsw (body_embedding vector_cosine_ops)
  WHERE tenant_id IS NOT NULL;

Three things to notice.

The vector is NOT NULL. It is not an optional projection of the row; it is part of the row's identity. A document without an embedding is not a row that exists yet. This is the same posture we take toward title or body: missing is not a state.

The model name and timestamp are on the row. When the embedding model changes — and it will — you do not need a separate audit table to know which rows still need re-embedding. The row tells you.

The HNSW index is co-located. It is built on the same table, in the same engine, by the same transaction that wrote the row. There is no separate index server, no separate replication lag for the index, no separate failover.

Transactions: write the text and the vector together

The producer code looks like one round trip, not three:

async function upsertDocument(input: DocumentInput): Promise<Document> {
  const embedding = await embedder.embed(input.body)

  const row = await reddb.tx(async (tx) => {
    return tx.upsert('documents', {
      id: input.id,
      tenant_id: input.tenantId,
      title: input.title,
      body: input.body,
      updated_at: new Date(),
      body_embedding: embedding,
      embedding_model: embedder.modelId,
      embedded_at: new Date(),
    })
  })

  return row
}

The embedding call is outside the transaction (it is the slow network hop). The write inside the transaction includes both the text and the vector. Either both are visible to the next reader, or neither is. The drift window is exactly zero.

Compare to the two-store version of the same operation, where you have to choose between:

writing the text first, then the embedding, and living with the window in between; or
writing the embedding first, then the text, and serving search results for a document that does not exist; or
writing to an outbox table inside the transaction, then having a downstream worker fan it out, and inheriting all the failure modes you were trying to avoid.

None of these are good. The third is the most defensible and it is still bad — the moment the worker is behind, you are back inside a drift window.

Reads: one query, not two

The retrieval side simplifies in proportion.

SELECT id, title, body, updated_at,
       1 - (body_embedding <=> $1) AS similarity
FROM documents
WHERE tenant_id = $2
  AND deleted_at IS NULL
ORDER BY body_embedding <=> $1
LIMIT $3;

One query. Tenant scoping and soft-delete filtering happen inside the same query that does the nearest-neighbor search, because they are columns on the same row. No second round trip to filter, no application-side join between "vector hits" and "current document state."

This is the line we keep coming back to: the two-store architecture forces the application to do a join the database could have done. The application is wrong about that join far more often than the database would be.

What chunking looks like when chunks are rows

Real documents do not embed cleanly as a single vector. You chunk them. The two-store world makes this a second table in the vector store with a foreign key back to the document — and a second drift window for the chunk-to-document relationship.

In the single-engine model, chunks are rows. They are children of the document:

CREATE TABLE document_chunks (
  id              UUID PRIMARY KEY,
  document_id     UUID NOT NULL REFERENCES documents(id) ON DELETE CASCADE,
  tenant_id       UUID NOT NULL,
  ordinal         INT NOT NULL,
  text            TEXT NOT NULL,
  embedding       VECTOR(1024) NOT NULL,
  embedding_model TEXT NOT NULL,
  embedded_at     TIMESTAMPTZ NOT NULL
);

CREATE INDEX chunks_embedding_idx
  ON document_chunks
  USING hnsw (embedding vector_cosine_ops);

ON DELETE CASCADE carries the chunks with the parent. When a document is deleted, its chunks vanish. When a document is rewritten, the upsert path deletes the old chunks and writes the new ones inside the same transaction:

await reddb.tx(async (tx) => {
  await tx.upsert('documents', { /* ...as above... */ })
  await tx.delete('document_chunks', { document_id: input.id })
  for (const [i, chunk] of newChunks.entries()) {
    await tx.insert('document_chunks', {
      id: crypto.randomUUID(),
      document_id: input.id,
      tenant_id: input.tenantId,
      ordinal: i,
      text: chunk.text,
      embedding: chunk.embedding,
      embedding_model: embedder.modelId,
      embedded_at: new Date(),
    })
  }
})

A reader hitting the database mid-transaction sees either the old set of chunks or the new one. Never a mix. Never a half-rewritten document. The retriever cannot return a chunk whose source paragraph has already been removed, because the chunk and the paragraph commit together.

What about re-embedding when the model changes?

The model will change. New embedding releases are quarterly events. Two-store architectures handle this with a forklift: stand up a parallel vector store, re-embed everything, swap. The forklift is its own outage class.

In the single-engine model, re-embedding is a background job that reads rows where embedding_model != <current>, recomputes the vector, and writes it back. Each row's update is a one-row transaction. Readers continue to use whichever model the row currently has. The transition is monotone: every commit moves one more row to the new model, no row is ever in an inconsistent state, and you can pause and resume the migration without thinking about it.

async function reEmbedBatch(modelId: string, batchSize = 100) {
  const stale = await reddb.query(
    `SELECT id, body FROM documents
       WHERE embedding_model != $1
       LIMIT $2 FOR UPDATE SKIP LOCKED`,
    [modelId, batchSize],
  )

  for (const row of stale) {
    const v = await embedder.embed(row.body)
    await reddb.update('documents', {
      where: { id: row.id },
      set: { body_embedding: v, embedding_model: modelId, embedded_at: new Date() },
    })
  }
}

FOR UPDATE SKIP LOCKED lets you fan this out across workers safely — each worker grabs a chunk of stale rows and updates them, no coordination needed.

What we did not solve

Some things are unchanged by collapsing the stores.

You still need a good embedding model. Co-locating the vector does not improve recall.

You still need to chunk. Naive whole-document embeddings perform worse than chunked ones for long-form material, and that is independent of where the vector lives.

You still need evaluation. A retrieval pipeline you do not measure is one you cannot improve. Evals belong on the same database — see our piece on storing eval datasets as rows — but the act of building them is yours.

You still need to think about cost. A VECTOR(1024) column is four kilobytes per row before quantization. At a hundred million rows, that is non-trivial storage. HNSW indexes have their own memory footprint. None of this is free; it is just no longer split across two budgets owned by two teams.

The one bug class we removed

We did not build a magic retrieval system. We removed an entire category of operational failure — the kind that lives in the seam between "document changed" and "vector reflects the change" — by removing the seam. There is no drift window because there is no second store to drift against. There is no reconciliation job because there is nothing to reconcile. There is no orphaned vector because deletion of a document deletes its embeddings inside the same transaction.

That is one bug class, not all of them. But it is the bug class that, in our experience, drives the most off-hours pages on production RAG systems. Trading a queue and a worker and a sidecar for a column on a row turns out to be a very good trade.

Per-row encryption with zero-downtime key rotation

2026-05-10

Most "field-level encryption" stories fall apart at rotation time. The cryptography is fine. The operations aren't. Rotating a master key shouldn't require a maintenance window, a feature freeze, or a frantic Slack thread when read latency triples because every row in a hot table is being decrypt-then-re-encrypt'd while traffic still hits it.

The vault layer in RedDB is built so rotation is a background operation that runs at whatever pace you tell it to, on a live table, with no read-path stall and no application changes. This post is the internals. If you operate the system, the second half is the one you want.

The envelope

Every encrypted row carries a small envelope alongside its ciphertext. Three fields:

+----+-----------------+------------+
| g  | iv (12 bytes)   | ciphertext |
+----+-----------------+------------+
  ^
  generation byte (or short int)

g — the master key generation this row was encrypted under. One byte is plenty; we use a 16-bit int to leave room.
iv — a per-row, never-reused initialisation vector. AES-GCM, so 12 bytes.
ciphertext — the encrypted payload, with the GCM auth tag appended.

The envelope is stored as one opaque column. Application code never touches it directly; the SQL layer wraps reads and writes with vault_decrypt(col) and vault_encrypt(col) calls that look up the right key by generation.

The crucial property: the row knows which key it needs. No central index, no separate "which-key-for-which-row" table. That's what makes rotation cheap.

The master key set

The master key set is versioned, not replaced. Conceptually:

generation │ key material (32 bytes)        │ state
─────────────────────────────────────────────────────
1          │ <kek-1, wrapped by KMS root>   │ retired
2          │ <kek-2, wrapped by KMS root>   │ active-read
3          │ <kek-3, wrapped by KMS root>   │ active-write

Three states matter:

active-write — the generation new writes use. Exactly one at a time.
active-read — generations that are still permitted on the read path. Many can be active-read.
retired — generation is gone. Any row still tagged with it is unreadable (this is by design — see "destroying a generation" below).

The actual key material is wrapped by your KMS root and only unwrapped into a process-local cache. The wrapped form is what we store and replicate; the cache holds unwrapped keys with a short TTL so a KMS access revocation propagates.

Reads pick the right generation

A read is straightforward:

Fetch the row. Parse the envelope header.
Look up generation g in the key cache. If absent, ask KMS to unwrap it; populate the cache.
Verify the GCM tag, decrypt, return plaintext.

Cost: a 16-bit read of g, a map lookup (hot), and one AES-GCM decrypt. The expensive part — KMS round-trip — only happens on cache miss, which is essentially never in steady state because the working set of live generations is tiny (usually 1–3).

If the generation is retired, the read errors with a typed KeyRevoked and the row is returned as encrypted bytes plus the generation number — useful for forensics and for the "I told you to retire it" conversation.

Writes always use the latest

Writes are simpler still: encrypt under whichever generation is active-write, write the envelope, done. There is no read-modify-write on the encryption path. Inserts and updates are symmetric.

The application gets identical SQL whether the column is vaulted or not:

insert into customers (id, email_vault)
values ($1, vault_encrypt($2));

select id, vault_decrypt(email_vault) as email
from customers
where id = $1;

vault_encrypt is a stable function that picks the current active-write generation per call. vault_decrypt reads the envelope's embedded generation, so the same query works across rows encrypted under different generations — which is the whole point.

Rotation, online

Rotation is three steps and the third one is the only slow one.

Step 1 — provision a new generation. Generate kek-N+1, wrap with the KMS root, store it as retired initially so it isn't picked up by anything. This is one round-trip to KMS and a single SQL write.

Step 2 — promote. Atomically:

Move the current active-write to active-read.
Move the new generation from retired to active-write.

From this instant, every new write uses the new generation. Every old row remains readable because its old generation is now active-read. The application sees nothing. No restart, no flush, no schema change. Read latency does not change. p99 does not move.

If you stop here, you have a partially-rotated table. That's actually fine — rotation is monotonic and incremental. Many compliance regimes only require "new writes use the new key from date X." If yours does, you're done.

Step 3 — background re-encrypt. A worker scans the table, decrypts each row under its old generation, re-encrypts under the new one, and writes it back as a normal update. We use SKIP LOCKED so the worker never blocks an application write, and a rate-limit knob (rows-per-second) so the worker can be tuned to fit available IO headroom.

-- worker query, roughly
with batch as (
  select id from customers
  where vault_generation(email_vault) < (select current_active_write_generation())
  order by id
  limit 1000
  for update skip locked
)
update customers c
   set email_vault = vault_reencrypt(c.email_vault)
  from batch b
 where c.id = b.id;

vault_reencrypt decrypts under the embedded generation and re-encrypts under the current active-write. It's a single round-trip to the vault layer, no application code involved.

Once the worker finishes, every row carries the latest generation. The old generation can be moved from active-read to retired, and the wrapped key can be deleted from KMS. Now rotation is complete.

What "zero downtime" actually means

Specifically:

No read-path stall — reads of old rows during the worker pass go through the normal decrypt path with the old key, which is still active-read.
No write-path serialisation — writes during the worker pass land under the new generation and don't conflict with the worker's updates beyond the usual row locks.
No application changes — generation handling is below the SQL layer.
No required maintenance window — the only "instantaneous" operation is the generation promotion in step 2, which is a single-row update on the key-set table.

What it does cost:

IO during the background pass. A re-encrypt is approximately one logical read + one logical write per row, plus WAL.
Cache memory for any live generation. Trivial.
KMS unwrap calls when a new generation appears or a cache TTL expires. Small.

The knob to leave alone

The vault has a rotation.aggressive flag that forces a full synchronous re-encrypt scan inside the transaction that promotes the new generation. It exists because the test suite needs to deterministically produce a fully-rotated table to assert on. It is catastrophic on a real workload:

It serialises every row in the table behind the promotion transaction.
It holds locks long enough that anything touching the table will queue.
On a multi-terabyte table it will not complete inside any reasonable transaction timeout, and you will get to find out what your client-side retry behaviour does when a 4-hour transaction aborts.

We've thought about removing it. Strong opinion: if you find yourself reaching for rotation.aggressive on a production cluster, the actual answer is to let the background worker run and check back in a few hours.

Destroying a generation

Real revocation — "make this key materially impossible to use, even by an operator with database root" — is the same dance from the opposite direction:

Move the doomed generation to retired in the key-set table.
Delete the wrapped key from KMS.
Any row still tagged with that generation is now permanently undecryptable.

Step 3 is the bit you actually want for incident response: even a leaked database backup is useless without the KMS-resident master key. The envelopes are still there; they just can't be unwrapped.

This is also how erasure works for compliance — for a "forget this user" request that crosses replicas, backups, and WAL archives, the cleanest path is per-tenant generations: retire and destroy the tenant's generation and every replica, backup, and WAL segment containing that tenant's ciphertext becomes cryptographically inert in one step. We'll write that up separately.

Where this lives in the architecture

The vault layer sits between SQL parsing and the storage engine. From the SQL layer's point of view, encrypted columns are just bytes with two helpful functions. From the storage engine's point of view, there's no encryption at all — it sees opaque blobs.

That separation is what makes the system honest about its threat model. We protect against backup theft, replica theft, and KMS-tier revocation. We do not protect against a compromised database process — the unwrapped keys are in memory by design while the system is serving traffic. If your threat model includes "operator with database root reads decrypted rows," you need a different layer of the stack (typically client-side encryption, where the database never sees plaintext or unwrapped keys).

The shorter version

Each row carries a generation byte.
Master keys are versioned, never replaced.
Reads use the row's generation; writes use the current active-write.
Rotation is: provision, promote, then background re-encrypt at your own pace.
The synchronous-re-encrypt flag exists for the test suite. Leave it alone.

The follow-ups in this series will cover per-tenant key isolation, BYOK and KMS integration patterns (and the failure modes nobody writes about), and GDPR right-to-erasure when your data is replicated, embedded, and backed up.

Notes from compaction hell

2026-05-06

LSM trees are a great default until they aren't. We spent a quarter pushing on RedDB's storage engine under a workload that turned out to be antagonistic to the defaults — and a quarter is roughly how long it took to learn which knobs mattered. This post is the version of those notes that we wish someone had handed us in week one.

The shape of the problem matters, so: the workload was mixed OLTP (~80% point reads, ~15% range scans, ~5% writes), values ranging from 200 bytes to 64 KiB with a long tail, and a working set roughly 3× larger than RAM. Steady-state throughput was fine. The pain was the latency tail during major compactions: P99 spikes from ~4 ms to ~180 ms, lasting 30–90 seconds, repeating every few hours.

The Grafana screenshot we kept staring at

Imagine a P99 latency panel with a flat green floor at 4 ms and four red mountains per hour reaching 180 ms. The mountains are not random — they line up exactly with lsm.compactions.in_progress going from 0 to 1, with disk.read.bytes_per_sec doubling, and with the OS page cache hit rate dropping 8 percentage points. The peaks themselves are not the worry; the duration is. A 30-second window of bad tail latency, four times an hour, means a non-trivial fraction of customer requests get caught.

The first three things we tried did nothing useful.

What did not help

Reaching for tiered compaction at the first sign of write amp. Tiered compaction trades read amp for write amp. Our workload was read-heavy. Switching strategies made the steady-state P50 worse without flattening the tail. We reverted in a week.

Tuning per-level fan-out without first looking at value sizes. We spent a sprint sweeping level_multiplier from 8 to 16 and the only meaningful signal was noise. The actual problem was that compaction was moving 16 KiB values around to keep the keys sorted, and the keys weren't the part that hurt.

Adding more compaction threads when the bottleneck was disk, not CPU. iostat -x 1 was showing the SSD pinned at 95% utilization during compactions; CPU was sitting at 30%. Throwing more threads at it made the tail worse because the threads contended for I/O queue depth that the device couldn't honor.

The lesson is dull but real: measure before turning knobs. We had Prometheus metrics for compaction count but not compaction byte volume, and not the breakdown between key and value bytes. Adding those was the single most valuable hour of the quarter.

What did help

1. Per-level bloom filter sizing

The default bloom configuration used 10 bits per key globally. That's a defensible default — false positive rate around 1% — but the L0/L1 levels are on the hot path for every point read, and the cost of a false positive there is a full SSTable read on the wrong file.

[lsm.bloom]
# 10 bits/key was the global default. Hot levels benefit from more.
level_0 = 16  # FPR ~0.04%, costs ~6 bits/key of RAM per level-0 file
level_1 = 14  # FPR ~0.16%
level_2 = 12  # FPR ~0.61%
default = 10  # FPR ~1%, for cold levels where bloom RAM matters more

Effect on P99 point-read latency at steady state: 4.2 ms → 3.1 ms. Effect on RAM for bloom filters: +18%. Worth it.

2. Separating large values from the key-ordered storage

The biggest single win. Anything over 8 KiB now gets written to a blob area and the SSTable only stores a pointer. Compaction stops moving cold image bytes around to keep numeric IDs in sorted order. Background write volume during major compaction dropped 4.7×.

The configuration:

[lsm.blob]
enabled = true
min_value_size_bytes = 8192
gc_threshold_ratio = 0.5  # rewrite blob file when 50% of its bytes are dead

Effect on P99 during compaction: 180 ms → 42 ms. This is the change that actually made the on-call complaints stop. If you remember nothing else from this post: blob separation for any workload with a meaningful tail of large values.

3. Backpressure on the write path

A slow client is uglier than a buggy one. When the LSM gets behind — when L0 has more files than level0_slowdown_writes_trigger — we now intentionally inject delay into write acknowledgements rather than letting L0 grow unbounded. Letting L0 grow unbounded leads to a cliff: eventually compaction can't keep up at all, reads start touching dozens of files, and the engine becomes unrecoverable without operator intervention.

[lsm.write_pressure]
level0_slowdown_writes_trigger = 20  # start adding delay
level0_stop_writes_trigger = 36      # hard stop, return WriteThrottled
soft_pending_compaction_bytes_limit = "64GiB"
hard_pending_compaction_bytes_limit = "256GiB"

The client sees a WriteThrottled error sooner under stress, which is a clear signal callers can react to (retry with backoff, queue, shed load). The alternative — silent unbounded growth until the engine collapses — is much worse for everyone.

4. Compaction priority by score, not by level

The default scheduler prefers compacting whichever level is most over-target. That's fine in isolation but it ignores the fact that some files have many more delete tombstones than others, and compacting those is a much better use of I/O budget than evenly trimming each level. We added a tombstone-density signal to the compaction priority score:

fn compaction_priority(file: &SSTable) -> f64 {
    let size_pressure = file.size_bytes as f64 / file.target_size_bytes as f64;
    let tombstone_factor = 1.0 + 3.0 * file.tombstone_ratio.min(1.0);
    let age_factor = 1.0 + 0.5 * (file.age_seconds() / 86_400.0).min(2.0);
    size_pressure * tombstone_factor * age_factor
}

Read amp on tables with heavy churn dropped noticeably. The exact gain depends on your write pattern, but for our workload — where a few hot tables had ~40% tombstone density — point reads on those tables went from touching 4–6 SSTables to typically 1–2.

Concrete numbers, before and after

The table below is steady-state during business hours, averaged across our three largest production clusters, with the same workload and the same hardware.

Metric	Before	After	Notes
Read P50	1.8 ms	1.4 ms	Mostly the blob change keeping the page cache useful
Read P99 (no compaction)	4.2 ms	3.1 ms	Per-level bloom sizing
Read P99 (during major compaction)	180 ms	42 ms	Blob separation is the bulk of this
Write P99	6.0 ms	5.4 ms	Mostly unchanged; backpressure rarely triggers
Bytes compacted per hour (peak)	24 GiB	5.1 GiB	Blob bytes don't move with keys anymore
RAM for bloom filters	1.1 GiB	1.3 GiB	The tax for the bloom change
`WriteThrottled` errors per day	0 (engine fell over instead)	~12	A clean error is better than a cliff

The P99-during-compaction row is the one we care about most. 42 ms is not great, but it does not generate customer-visible incidents. 180 ms did.

Tiered compaction: revisited, and still wrong for us

After the other changes landed, we tried tiered again. With blob separation pulling the large-value cost out of compaction, the write-amp argument for tiered is much weaker. With per-level bloom sizing, the read-amp penalty of tiered is more visible because the rest of the system is now faster. Tiered lost worse the second time than the first.

If your workload is write-dominated with small values, the conclusion is probably opposite. We didn't have that workload, so we don't have credible numbers for it.

What we still don't love

The latency tail during major compactions is honest, not pretty. 42 ms P99 is a number we can live with operationally — it doesn't trip SLOs — but it's a number, not a zero. The next move on our list is a "compaction budget" that limits how much I/O bandwidth compaction is allowed to consume during business hours and lets it catch up overnight. That's a different post; we haven't shipped it yet.

The other thing we haven't solved: snapshot isolation under heavy compaction load is fine for correctness but causes a small extra read amp because old SSTable generations have to stick around for readers. We've measured it at ~3% of total read I/O. It's the kind of thing you only see if you go looking.

The one knob to turn first

If you remember one configuration change from this post, make it blob separation. Every workload we have ever profiled benefits, and the ones with even a small fraction of large values benefit dramatically. The other three changes are worth doing, but they are increments. Blob separation is the cliff.

The order we wish we had done it in: blob separation, observability for compaction byte volume by file, per-level bloom sizing, backpressure thresholds, score-based priority. We did it roughly backward, which is why it took a quarter instead of a sprint.

When NOT to migrate: a checklist

2026-05-04

Most migration posts are sales pitches dressed up as advice. This one is the opposite. We have helped four teams migrate off Firebase and three off MongoDB in the last twelve months, and the most useful conversation we have with prospective customers is the one where we tell them not to migrate yet.

A migration is a project that has a real shape: weeks of senior-engineer time, an integration risk window, a regression bug or two that ships to production, and a quarter of feature work you did not do because you were busy moving data. If the pain you are fleeing is smaller than that, you are about to trade a known annoyance for an unknown one. So before you cut, run the checks below. Each one ends in a "stay" verdict if the answer is honest. We have written it as a checklist on purpose — if you can put a tick next to every "stay" signal, close this tab and go back to the feature you were building.

Check 1: have you actually quantified the pain?

Engineers migrate because something feels slow, expensive, or fragile. "Feels" is not a metric. Before you spend a quarter on a migration, the following numbers should be on a shared doc:

The actual dollar cost per month, broken out by line item. Not "Firebase is expensive" — "Firestore reads are $4,800/mo and growing 18% MoM."
The actual P99 latency of the queries that hurt. Not "Mongo is slow" — "the $lookup pipeline behind /feed runs in 1.4s P99 and we have a 300ms target."
The actual incident count attributable to the current stack in the last 90 days. Not "we have ops pain" — "three Sev-2s in 90 days, two of which were rooted in MongoDB primary failover behaviour."

If you cannot put numbers in those three slots, your pain is vibes. Stay. Spend the next two weeks instrumenting instead. Half the time the numbers come back smaller than the team's intuition, and the migration cancels itself.

Check 2: how big is the team?

Migrations have a U-shaped cost-by-team-size curve. The two-person team can move fast (no coordination tax) but cannot afford to lose either engineer for four weeks. The fifty-person team has the slack but pays an enormous coordination cost (every team owns part of the schema). The teams that suffer least are in the awkward middle — five to fifteen engineers — and that is also where most of our customers sit.

Team size	Migration mode that works	What to watch out for
1–2 engineers	Cut over in a sprint, no dual-write phase	One bad incident kills the company. Have a rollback plan you have actually tested
3–5 engineers	Dual-write for two weeks, then promote	The engineer doing the migration cannot also be on-call. Pre-negotiate this with the team
6–15 engineers	Dual-write for a month, feature-flag the read side	The migration owner needs explicit air cover from the CTO. Otherwise feature work eats the schedule
16+ engineers	Per-service migration over a quarter	Every service owner thinks their slice is special. Budget time for the meetings

Stay if your team is at the edges (1 or 16+) and the pain numbers from Check 1 are not severe. The smallest teams should stretch the existing stack until either revenue or a hire makes a migration safer; the largest teams should plan a per-service migration as a multi-quarter program, not a project.

Check 3: is the pain in the data layer or somewhere else?

This is the check we wish more teams ran first. "The database is slow" turns out, surprisingly often, to be "our N+1 query in the orders service is slow" or "the cache layer is bypassed for the homepage." Run an honest five-day investigation before you blame the storage engine:

Profile the worst three endpoints. Where is the latency actually accumulating?
Look at the query plans for the worst three queries. Are they doing what you expect?
Check whether the most expensive endpoints have working caches and whether the caches are actually hit.
Audit the slow-query log for the last week. Are 80% of the slow queries from a handful of identifiable shapes?

If the answer is "yes, it is a small number of fixable shapes" — stay. A query rewrite is cheaper than a migration, even when the migration eventually arrives anyway. We have watched teams migrate to "fix" a Mongo problem that turned out to be an unindexed field; they shipped the same N+1 query on the new engine, and the new engine ran it slowly too.

Check 4: what is your runway?

Migrations consume runway. Concretely:

Senior-engineer-weeks: figure 4–10 depending on dual-write scope.
Reduced feature velocity for the rest of the team during the cutover window (people get pulled into adjacent reviews and incident response).
A higher-than-baseline rate of regressions in the first 30 days after cutover.

If your company will run out of money inside that window, stay. If the pain from Check 1 is "we'll be insolvent if Firestore bills keep growing," consider a partial migration — move only the worst-cost path to a cheaper store, do not rewrite the auth path or the document model. Partial migrations sound ugly on the architecture diagram and work fine in practice.

Check 5: do you have a tested rollback?

Every migration plan promises rollback. Roughly one in four actually has one that has been exercised. Before you start, write the runbook for "the new engine is on fire, we need to be back on the old one in two hours." Read it back to yourself. Can you actually do it?

The honest version of rollback looks like:

Dual-write was on the whole time, so the old store still has correct data through last write.
Feature flag flips reads back to the old store.
Whatever writes happened only on the new store between cutover and rollback are surfaced for manual reconciliation (a script lists them; a human reviews).
The team has actually run this end-to-end on staging within the last two weeks.

If steps 1–4 are not all true, stay. Tighten the rollback story until they are. The cost of doing this work is much smaller than the cost of an unplanned rollback at 3am, and roughly the same as the cost of an unplanned non-rollback (the case where rollback was possible in principle but nobody dared push the button).

Check 6: is the new stack actually better for your shape?

This is where we will be most direct against our own product. RedDB is good at workloads with cross-shape access patterns (RAG over mutable docs, agent memory, search-over-application-data, multi-tenant SaaS where you need vector + text + structured per tenant). It is not the right choice for:

Single-shape workloads. If you only need OLTP, Postgres is more mature and has a larger labour market. Stay on Postgres.
Analytics-dominant workloads. If 80% of your queries are scans over historical data, ClickHouse or DuckDB will beat us. Stay on what you have until you can move to a column store.
Embedded / single-process workloads. SQLite is the right answer for a desktop app or a CLI. We are an over-investment.
Hyper-low write volume. If your bill on the current managed Postgres is $40/mo, the operational overhead of a less-mature engine is the bigger cost. Stay on the managed service.

We wrote a whole post on this — When you don't need RedDB — and the short version is: if the migration target is not unambiguously better for your specific workload shape, the migration is paying for an answer to a question you did not ask.

Check 7: who on the team will own the new stack at 3am?

This is the social check, and it kills more migrations than any technical one. Migrations frequently succeed for six months and then quietly fail in month nine when the engineer who ran the migration leaves the company. The new stack is now an artifact nobody understands; on-call rotations get awkward; the next migration is back to the old engine.

Before you migrate, name on a shared doc:

The primary owner of the new stack.
The secondary owner.
The cross-training plan to get to at least three people who can debug it under pressure.
The runbook location.
The dashboards.

If you cannot fill in those slots with real names, stay. Hire or train first.

The "go" signal looks like this

After all seven checks, you should migrate only if all of these are true:

The pain has real numbers attached, and the numbers are worse than the migration cost.
The team size matches a viable migration mode (or you can wait until it does).
The pain is genuinely in the data layer, not somewhere upstream.
The runway covers the migration window with a buffer.
A rollback plan exists and has been exercised on staging.
The new stack is unambiguously better for your specific workload shape.
At least two engineers will own the new stack with names on a doc.

If even one of those is false, the cheapest move is "stay, fix the highest-leverage thing on the current stack, revisit in a quarter." We tell prospective customers this often enough that we have a stock phrase for it: the best migration is the one you did not have to do this quarter.

TL;DR

Migrations look like architecture work and behave like organisational work. The technical part is comparatively easy; the runway, ownership, and rollback parts are what kill projects. Before you cut over, write down the numbers (Check 1), match the team size to a viable mode (Check 2), make sure the pain is actually in the data layer (Check 3), confirm runway (Check 4), exercise rollback (Check 5), check the new stack fits your shape (Check 6), and name the people who will own it (Check 7).

If you tick every box, run the migration. If not, the Out of Firebase and Mongo → RedDB posts will still be here in six months when the answer is yes.

In the meantime: instrument, profile, rewrite the worst queries, fix the cache, and bank the engineer-weeks. That is the boring answer and it is right more often than the exciting one.

GDPR right-to-erasure when your data is replicated and embedded

2026-05-03

A user clicks delete my account. Forty seconds later their row is gone from the primary. The DPO signs the ticket. Eleven months later a vendor audit pulls a backup, replays it into a sandbox, and the deleted user's chat history shows up in a similarity search against an embedding index that nobody remembered to rebuild. The audit goes from clean to findings-with-fine-exposure on a single query.

GDPR Article 17 sounds like a DELETE FROM users WHERE id = ?. In a system with replicas, derived embeddings, time-travel backups, and a WAL the regulator can subpoena, it is not. This is the operator playbook for what RedDB does for you, what it does not, and the holes you have to close by hand.

The four places PII hides after `DELETE`

Before discussing tooling, name the surfaces. Every system has them; most teams have only enumerated two:

Live row — the obvious one. DELETE removes it from the table.
Replicas and read followers — async replication eventually catches up. The window is usually seconds, but during a partition it can be minutes.
Derived data — anything computed from the row that wasn't itself the row. Embeddings are the headline case. Materialised views, denormalised search indexes, ML feature stores, downstream warehouses, cached snapshots in your CDN — all of these.
Time-travel artefacts — backups, WAL segments retained for PITR, snapshot exports shipped to a DR region. These are intended to survive deletes. That is exactly the problem.

Regulators have caught up to (3). The 2024 EDPB opinion on AI training data, the Hamburg DPA's 2025 ruling on embedding personalisation, and the French CNIL's guidance on vector stores all say the same thing in different words: a vector derived from personal data is personal data. You cannot point at an opaque float array and call it anonymous. If the vector lets you identify the subject through similarity search, it counts.

This is why your erasure workflow now has to traverse the derived layer, not just the source.

What RedDB does automatically

In order of how confident you should be:

Hard DELETE on the live row. The row is gone from the table. Foreign-keyed children with ON DELETE CASCADE are gone too. This is table-stakes and works.
Replication propagates the delete. Logical replication carries the tombstone to followers within the standard lag window. The replication_lag_bytes metric (see the self-host checklist) tells you when to wait. There is no separate replica-erasure command; the same WAL record that deletes on the leader deletes on the follower.
Cascading delete of embedding columns. This is the payoff for keeping the embedding on the row instead of in a second store. When the row goes, the vector goes — atomically, in the same transaction. There is no eventual job, no queue, no drift window. Compare with the multi-store shape, where you delete from Postgres and then have to remember to delete from Pinecone or Qdrant, and where a botched second call leaves the embedding behind.
Background compaction of the on-disk tombstone. The LSM compactor (see compaction notes) eventually rewrites SSTables and the deleted bytes stop existing on disk. Typical worst-case 24–48 hours under steady write load; faster under explicit reddbctl compact --tombstones.

That's the floor. Most teams stop here, ship a "we are GDPR-compliant" page, and find out about the rest during an audit.

What still needs an operator runbook

The three surfaces RedDB does not erase automatically:

Surface 1 — Backups and WAL segments retained for PITR

Your backup policy probably retains daily snapshots for 30 days and WAL for 35 (the numbers from the DR post). For 35 days after the user clicks delete, the previous state — including their PII — is sitting in object storage.

Three strategies, in order of how regulators actually treat them:

(a) Wait it out — "deletion by expiry." Most DPAs accept that a backup taken before the erasure request, retained for a documented and justified retention period, will eventually rotate out. You document: "user deleted at T; backups expire at T+30d; full erasure complete at T+35d." The audit looks for the policy and the expiry receipt, not real-time scrubbing.

This is the easy answer, but only if your retention is short and bounded. If you keep yearly backups "for accounting" you are signing yourself up for surface 1 problems forever.

(b) Crypto-shredding. This is why the per-row encryption and BYOK/KMS work matters. If every tenant — or every user, if you go that far — has a distinct DEK wrapped by a CMK, deleting the CMK makes every backup of their data ciphertext-only forever. You do not need to rewrite the backups; the key to read them no longer exists.

The operator command for this looks like:

# Schedule the per-tenant CMK for deletion in your KMS.
# Use the longest pending-window your KMS allows so accidental
# triggers are recoverable, but make the window expiry mechanical.
aws kms schedule-key-deletion \
  --key-id "$TENANT_CMK_ARN" \
  --pending-window-in-days 7

# Record the scheduled deletion against the erasure ticket.
reddbctl audit log \
  --event "erasure.crypto-shred.scheduled" \
  --tenant "$TENANT_ID" \
  --cmk "$TENANT_CMK_ARN" \
  --completion-at "$(date -d '+7 days' --iso-8601)"

Crypto-shredding is the only approach that handles backups and WAL segments and DR replicas in one operation. It is also the one regulators have started naming approvingly in guidance (CNIL 2024, ICO 2025). Cost: you must have built per-subject DEK separation up-front. Retrofitting it is months of work.

(c) Backup rewrite. Restore each backup, run the delete, re-snapshot, encrypt, ship to the same object-store path. Possible. Slow. Expensive. Most teams who try this once switch to (b).

Surface 2 — Derived embeddings that aren't a column on the row

The atomic-delete-via-cascade story only works if the embedding lives on the row. If you copied embeddings into:

a separate vector store (Pinecone, Qdrant, Weaviate, pgvector in a different cluster),
a feature store keyed by user_id,
a search index that retained the chunked text,
a downstream warehouse fact table,

…then DELETE FROM users does nothing to those copies. You need an explicit fan-out.

The pattern is an erasure outbox. Same row, same transaction:

BEGIN;

DELETE FROM users WHERE id = $1;

INSERT INTO erasure_outbox (subject_id, surfaces, requested_at)
VALUES (
  $1,
  ARRAY['vector_store_v2', 'feature_store', 'warehouse_pii_dim', 'cdn_cache'],
  now()
);

COMMIT;

A worker drains the outbox and issues the surface-specific delete to each downstream system. The outbox row is only removed once each surface has confirmed (or returned 404, which counts). The DPO dashboard joins erasure_outbox against the ticketing system: any subject older than the legally-binding deadline (30 days in the EU, 45 in California, less under many sectoral rules) and not fully fanned-out is the on-call's problem.

The transaction guarantees you cannot delete the live row without enqueueing the downstream sweep. The opposite mistake — sweeping downstream and forgetting to delete locally — is impossible because the delete and the enqueue are in the same WAL record.

Surface 3 — Embeddings as PII even when the row is gone

This is the surface most teams discover during their first regulator inquiry. You followed the cascade pattern, the embedding column went with the row, you can prove it. Then the auditor asks: "what about the embedding's position in the index? You have an HNSW graph. Nearest-neighbour queries against the index still return useful information about the deleted subject for some hours until compaction rebuilds the graph."

HNSW graphs are an interesting case. Deleting a vector marks the node as soft-deleted; queries skip it; but the graph topology — the links — were chosen partly because of that vector's position. The information leakage from this is microscopic and academic. Regulators have not asked about it in any case we've heard of. But it is the kind of thing a determined auditor will raise, so:

The reddbctl index rebuild --tombstone-aware vec_index_users command (current RedDB) fully reconstructs the graph from live vectors. Run it monthly or after any bulk erasure event.
Document the rebuild cadence in your DPIA so it does not become a finding.

A complete erasure runbook

Stitching it together, this is what a production erasure looks like end-to-end:

-- T+0: subject requests erasure via support portal.
-- Ticket system writes a row to erasure_requests with a 30-day SLA.

BEGIN;

-- 1. Hard delete from primary, cascading to chunks + embedding column.
DELETE FROM users WHERE id = $subject_id;

-- 2. Enqueue downstream fan-out. Same transaction = no drift.
INSERT INTO erasure_outbox (subject_id, surfaces, requested_at)
VALUES ($subject_id, ARRAY['vector_store_external', 'warehouse', 'cdn'], now());

-- 3. Audit log entry that satisfies Article 30 record-keeping.
INSERT INTO audit_log (event, subject_id, actor, at)
VALUES ('erasure.requested', $subject_id, $support_agent_id, now());

COMMIT;

# T+seconds: replication catches up. Watch the metric.
reddbctl metric replication_lag_bytes --threshold 0 --wait 60s

# T+minutes: outbox worker fans out.
# Each surface's DELETE is idempotent; 404 is success.
# Outbox row removed only when every surface returns success.

# T+24-48h: LSM compaction rewrites SSTables, on-disk bytes gone.
reddbctl compact --tombstones --table users  # optional acceleration

# T+7d: crypto-shred CMK if per-subject keys.
aws kms schedule-key-deletion --key-id "$SUBJECT_CMK" --pending-window-in-days 7

# T+30d: snapshot retention boundary. Last snapshot containing the
# subject expires. Erasure complete by policy.

# T+monthly: HNSW graph rebuild absorbs any soft-deleted vectors.
reddbctl index rebuild --tombstone-aware vec_index_users

The DPO sees a single dashboard: erasure ticket id, T+0, current surface status, completion ETA against the 30-day legal deadline.

What's automatic vs. operator-driven (the cheat sheet)

Surface	RedDB does	You have to do
Live row	`DELETE` cascades to children and embedding column	—
Replicas / followers	Tombstone replicated via WAL	Verify `replication_lag_bytes` returns to 0 before closing ticket
On-disk tombstone	Compaction reclaims bytes on standard schedule	(Optional) `reddbctl compact --tombstones` to accelerate
Backups (snapshot)	Expire on retention schedule	Document policy; consider crypto-shredding for tighter SLA
WAL retained for PITR	Expires with the segment	Crypto-shred for sub-retention-window deletion
External vector store	—	Fan out via `erasure_outbox`
Warehouse / feature store	—	Fan out via `erasure_outbox`
HNSW graph topology	Soft-deletes from query results	Monthly rebuild with `--tombstone-aware`
Audit log of the erasure itself	Append-only event	Keep this — Article 30 requires it

The right way to read this table: anything in the "you have to do" column is the audit-finding column if you forget it. None of it is hard. All of it is operationally invisible to anyone who hasn't asked.

Three things that aren't in scope (yet)

Honesty matters. RedDB does not currently:

Surface a single reddbctl erase --subject ... command that drives the whole pipeline. That's a roadmap item — the integration points are too tenant-specific (which downstream surfaces, which KMS, which ticketing system) to ship a default that doesn't lie. For now, the runbook above is the contract.
Encrypt the audit log itself per subject. The audit row stating we erased subject X at time T is itself technically PII. Most DPAs accept this under Article 17(3)(b)/(e) (compliance with legal obligation and defence of legal claims). We aren't applying crypto-shredding to the audit log; you shouldn't either.
Sync the rebuild across regions atomically. If you maintain HNSW indexes in two regions, the rebuild runs region-by-region. The leakage window between regions is on the order of the rebuild duration (minutes to hours). Document it.

When to start

The fastest way to fall behind on Article 17 is to leave it for the audit. Ten engineering hours now — schema for the outbox, a worker, a dashboard, a runbook — costs less than ten hours of legal time after a single inquiry, even before any potential penalty.

The harder change — per-subject DEKs so that crypto-shredding is on the table — is months of work and only worth it if you sell to buyers who will ask. If you do not yet have those buyers, the policy-expiry approach (a) above is enough. The day a regulated buyer says we need crypto-shred, you start. Not before.

The one thing everyone should do today: write the outbox table and the fan-out worker. Cascading the embedding column on DELETE is the cheapest piece of Article 17 alignment you will ever buy, and most teams already structured their schema to support it without realising they were halfway there.

TL;DR

Article 17 is no longer just DELETE FROM users. Embeddings derived from PII are PII.
Keeping the embedding on the row turns the hardest erasure surface into a CASCADE.
Replicas erase by WAL; on-disk bytes by compaction; backups by retention or crypto-shred.
External surfaces (vector stores, warehouses, feature stores) need an erasure outbox driven in the same transaction as the delete.
HNSW graph topology is a tiny leakage surface; monthly tombstone-aware rebuild closes it.
Crypto-shredding via per-tenant or per-subject CMKs is the only single-step approach for backups and WAL. It only works if you architected for it.
The audit log of the erasure itself is allowed to persist under Article 17(3).

Snapshot isolation, in practice

2026-05-02

Saying "we use snapshot isolation" is a sentence. Living with it is a stack of small decisions that every team makes again, badly, the first time a customer hits a write-skew bug at 2am.

This post is the cheat sheet we wished we had: what you get, what you don't, what error your client actually sees, why we don't auto-retry, and the one application pattern that resolves most real-world write-skew without coordination.

What snapshot isolation gives you

Each transaction sees a consistent snapshot of the database as of the moment it began. Readers never block writers. Writers never block readers. Two transactions can read overlapping rows in parallel — neither waits for the other to commit.

For 90% of OLTP code that previously ran on read-committed Postgres or MySQL with REPEATABLE READ, this is a strict upgrade:

No more dirty reads, non-repeatable reads, or phantom reads inside a single transaction.
Long-running reports stop blocking the write path.
"Why is my SELECT seeing half-applied changes?" disappears as a category.

That part is real and wonderful. You can stop reading here if your workload is small and you only ever update one row per transaction.

What snapshot isolation does NOT give you

Snapshot isolation is not serializable. There's exactly one anomaly that survives, and it has a name: write skew.

The textbook example uses doctors going off-call. Forget the textbook. Here's one we actually shipped a patch for:

Real write-skew: tenant seat allocation

A SaaS plan permits up to N active seats per tenant. The check-and-insert looks like this:

BEGIN;
  -- T1 and T2 both see active_count = 4, max = 5
  SELECT count(*) FROM seats
   WHERE tenant_id = $1 AND active = true;
  -- T1 inserts seat #5 → 4 + 1 = 5, within limit, commit.
  -- T2 inserts seat #5 → 4 + 1 = 5, within limit, commit.
  INSERT INTO seats(tenant_id, user_id, active) VALUES ($1, $2, true);
COMMIT;

Both transactions read the same snapshot. Neither modifies a row the other modifies — they each insert a different new row. Snapshot isolation has no conflict to detect. Both commit. The tenant now has 6 active seats on a 5-seat plan.

Under serializable isolation the database would force one of them to abort. Under snapshot isolation, you have to do the work.

We've seen this exact shape in:

Inventory reservation (last unit sold twice).
Bank-style transfers when the constraint is "account balance must stay non-negative" and two debits land on the same account from different sessions.
Approval workflows where exactly one approver should advance a step.

The pattern is always: the invariant spans rows the transaction reads but does not modify.

What clients actually see

When two transactions modify the same row and one commits first, the second receives:

ERROR: serialization conflict on commit
SQLSTATE: 40001
DETAIL: write conflict with concurrent transaction <txid>

SQLSTATE 40001 is the standard signal. Every mature client library checks for it. We deliberately picked the same code Postgres uses so that existing retry middleware in your stack just works.

What the second transaction sees on its COMMIT call:

try {
  await db.transaction(async (tx) => {
    await tx.query('UPDATE counters SET n = n + 1 WHERE id = $1', [id])
    await tx.query('INSERT INTO events(...) VALUES (...)')
  })
} catch (err) {
  if (err.code === '40001') {
    // Conflict. Application decides what to do.
  }
  throw err
}

The whole transaction is rolled back. No partial state, no half-applied writes, no need to clean up manually.

Why we don't auto-retry

Several engines retry 40001 transparently inside the driver. We considered it. We chose not to. Three reasons:

Retries hide concurrency bugs. A function that silently runs 1–N times changes meaning. Idempotency stops being optional — it becomes a load-bearing requirement that nobody documents until production breaks.
The retry budget belongs to the application. "Retry 3 times with backoff" is correct for some workloads. For others (financial), the right answer is "abort, surface to user, let them re-issue." We can't pick that for you.
Auto-retry composes badly with side effects. If your transaction calls sendEmail() mid-flight via a trigger or hook, retry sends two emails. The retry decision has to live at the layer that knows what's safe to repeat.

So you get an explicit 40001 and you decide. In practice most teams add a small wrapper:

async function withRetry<T>(fn: () => Promise<T>, attempts = 3): Promise<T> {
  for (let i = 0; i < attempts; i++) {
    try { return await fn() } catch (err: any) {
      if (err.code !== '40001' || i === attempts - 1) throw err
      await sleep(20 * Math.pow(2, i) + Math.random() * 20)
    }
  }
  throw new Error('unreachable')
}

That belongs in your application layer. We refuse to put it in the driver.

The one pattern that fixes most real write-skew

For the seat-allocation example above, the cleanest fix isn't "switch to serializable" (which would force aborts on workloads that don't actually conflict). It's read-then-write CAS: include the value you read in the predicate of the write, so a concurrent change breaks your assumption explicitly.

For the seat example we keep a counter row per tenant:

CREATE TABLE tenant_seat_counters (
  tenant_id  uuid primary key,
  active_n   int  not null,
  version    int  not null default 0
);

The transaction becomes:

BEGIN;
  -- 1. Read the counter and version inside the transaction snapshot.
  SELECT active_n, version FROM tenant_seat_counters
   WHERE tenant_id = $1;

  -- 2. Check the invariant in application code.
  -- If active_n + 1 > plan.max_seats, abort and return a friendly error.

  -- 3. CAS the counter: predicate includes the version you read.
  UPDATE tenant_seat_counters
     SET active_n = active_n + 1, version = version + 1
   WHERE tenant_id = $1 AND version = $2;
  -- If rowcount = 0, another transaction won. Abort.

  -- 4. Insert the seat row.
  INSERT INTO seats(tenant_id, user_id, active) VALUES ($1, $3, true);
COMMIT;

Now both T1 and T2 race on the same row (the counter). Snapshot isolation detects the conflict on step 3 and aborts the loser with 40001. The application retries (or surfaces a "couldn't grab a seat, please try again" message). The invariant holds.

This pattern generalises: any time your invariant spans multiple rows, find or invent a single row whose version represents that invariant, and gate the write on that version. You've turned a write-skew anomaly into a write-write conflict, which snapshot isolation handles natively.

When read-then-write CAS isn't enough

A few real workloads escape this pattern:

Multi-tenant invariants that span an unbounded set of rows ("no two appointments overlap for the same room"). A version row works only if all writers consent to touch it; if you have legacy writers, you need range locks.
Cross-table invariants that traverse foreign keys with cascading effects. Here we suggest a per-aggregate version row owned by the aggregate root.
Hot counters under sustained contention. The CAS pattern becomes a thrash. Switch to a sharded counter (N rows, randomly chosen on write, summed on read) and accept eventual consistency in the count.

For the rare cases where none of the above works, we expose SET TRANSACTION ISOLATION LEVEL SERIALIZABLE. It costs more, aborts more, but kills write skew at the database layer. Use it surgically.

A quick checklist before you ship

If you're about to deploy a feature whose correctness depends on a multi-row invariant under concurrent writers, ask:

What's the invariant in one sentence?
Which rows do I read to verify it, and which rows do I write to enforce it?
If the read set and write set don't overlap, where does write skew hide?
Is there a single row whose version represents the invariant — and if not, can I add one?
If conflict-on-commit happens, what's the user-visible message?

We've never regretted answering those before merging. We've regretted skipping them more than once.

TL;DR

Snapshot isolation removes 90% of concurrency footguns and is the right default.
The 10% it leaves behind is write skew, and it's a real bug, not a textbook one.
Conflicts surface as SQLSTATE 40001. We don't auto-retry — that's an application decision.
For most write-skew, read-then-write CAS on a version row restores correctness with no global locking.
When CAS doesn't fit, switch the offending transaction to serializable. Pay the cost where it's earned.

Snapshot isolation isn't a slogan. It's a contract with a known asterisk. Read the asterisk before you sign.

BYOK, KMS, and the boring parts of multi-tenant secrets

2026-05-01

The vault post covered how rotation works when the master key lives inside RedDB's own keystore. That's fine for single-tenant deployments. The moment you sell to a regulated buyer — banks, healthcare, anyone with a CISO — they ask the same question: can we keep our own key? The answer has to be yes, and the integration has to survive the failure modes that BYOK quietly introduces.

This post is the boring half of the story. Configuration, failure modes, and the operational shape that keeps you from waking up to a stuck write path.

What BYOK actually means

"Bring your own key" gets used loosely. In practice there are three shapes:

Customer-managed KMS key (CMK). The customer creates a key in their cloud account; you call their KMS with an IAM role they grant you. AWS KMS, GCP KMS, Azure Key Vault — all the same shape. Most common.
External key store / HSM-backed. The CMK in KMS is itself wrapped by a key that lives in a customer-controlled HSM (CloudHSM, GCP EKM, Thales/Entrust on-prem). Every KMS operation hits the HSM. Higher latency, higher assurance, much higher operational tax.
Hold-your-own-key (HYOK). The customer's HSM is the only place the key exists; you never see it unwrapped. Architecturally this means the data either decrypts inside their network (proxy model) or doesn't decrypt at all in your hot path (envelope-with-remote-unwrap, see below). HYOK is rare and almost always vendor-specific.

RedDB supports (1) and (2) directly. (3) is a custom integration we've done twice; the failure modes are similar but worse.

For the rest of this post, "KMS" means the customer's chosen provider — AWS, GCP, Azure, or HSM-backed equivalent. They differ in API surface but not in the operational story.

The envelope, revisited

Recap from the vault post: each encrypted row carries an envelope (g, iv, ciphertext) where g selects a master key generation. In single-tenant mode, those master keys are wrapped by RedDB's own keystore.

In BYOK mode, the same generation byte selects a data encryption key (DEK) that is wrapped by the customer's KMS:

                                customer's KMS
                                    │
            ┌──── unwrap(EncryptedDEK_g) ────┐
            ▼                                │
       DEK_g (in memory, cached)             │
            ▼                                │
   AES-GCM(row.iv, plaintext) ─► ciphertext  │
                                             │
            stored on disk:                  │
       (g, iv, ciphertext, kms_arn)          │
                                             ▼
                                customer's KMS audit log
                                (one entry per unwrap)

Three things matter and people forget:

The DEK is never written unwrapped. Only the KMS-wrapped form (EncryptedDEK_g) is on disk, in a small vault_keyring table.
Unwrap is cached. Calling KMS on every row read would put 5–50ms of KMS latency on the hot path. The DEK lives in process-local memory with a TTL.
The KMS ARN/resource ID is recorded with the envelope. This matters for multi-tenant setups where different tenants use different CMKs (see "Multi-tenant" below).

Configuration

A minimal BYOK config for AWS KMS:

vault:
  provider: aws-kms
  region: us-east-1
  cmk: arn:aws:kms:us-east-1:111122223333:key/abcd1234-...
  iam_role: arn:aws:iam::111122223333:role/RedDBVaultAccess
  dek_cache:
    ttl: 1h
    max_entries: 1024
  kms_timeout: 2s
  kms_retry:
    attempts: 3
    backoff: exponential
    max_total: 5s

GCP is similar with kms_key: projects/.../locations/.../keyRings/.../cryptoKeys/... and a service account; Azure uses vault_uri and a managed identity. The shape is intentionally provider-shaped, not least-common-denominator — providers diverge in attestation, audit, and rotation semantics, and a fake-uniform abstraction would lie about them.

The four knobs that matter operationally:

Knob	Default	What happens if you get it wrong
`dek_cache.ttl`	`1h`	Too long: revocation slow. Too short: KMS request rate goes up 100×.
`kms_timeout`	`2s`	Too short: transient KMS hiccups cause write failures.
`kms_retry.max_total`	`5s`	Total wall-clock budget for unwrap including retries.
`dek_cache.max_entries`	`1024`	Sized for active generations × tenants. Default fits ~10 tenants.

dek_cache.ttl is the one that decides the security-vs-availability tradeoff. We default to one hour because most customers' "rotate immediately when an engineer leaves" runbook tolerates an hour of stale-key exposure. Some banks require five minutes. The blast radius of getting this wrong is one knob away — write it down in your runbook.

Multi-tenant: one CMK per tenant

The pattern most SaaS deployments want:

One CMK per customer, in their AWS account.
An IAM trust policy that lets RedDB assume a role into their account.
A row-level tenant_id column that selects which CMK was used to wrap the DEK for that row's generation.

The vault keyring table grows a tenant dimension:

CREATE TABLE vault_keyring (
  tenant_id     uuid NOT NULL,
  generation    smallint NOT NULL,
  kms_arn       text NOT NULL,
  wrapped_dek   bytea NOT NULL,
  created_at    timestamptz NOT NULL,
  retired_at    timestamptz,
  PRIMARY KEY (tenant_id, generation)
);

Reads look up (tenant_id, generation), unwrap once via the tenant's KMS, cache the DEK keyed on the same pair. Writes always use the tenant's active-write generation.

Two things this gets you for free:

Cryptographic deletion per tenant. Customer churns? Revoke their CMK in their KMS console. Every row of theirs becomes unreadable, instantly, with no scan over your storage. The bytes are still on your disks, but they're effectively shredded. This is the GDPR erasure shape your privacy lawyer wants.
No cross-tenant blast radius. A misconfigured IAM policy on Tenant A's CMK doesn't take down Tenant B's reads.

Two things it costs you:

Cache pressure scales with active tenants. Size dek_cache.max_entries accordingly. With 10,000 tenants and 8KB per DEK entry, that's 80MB of cache — fine. With 1M tenants, you're rebuilding the cache strategy. We have customers doing both; the 1M case uses a two-tier cache with an L2 backed by RedDB itself (yes, the database caches its own keys).
KMS rate limits per account. AWS KMS defaults to 10,000 RPS per region per account. If your tenants share an AWS account (rare) or your cache TTL is too low (common mistake), you'll hit it. Get the limit raised before launch, not during.

The three failure modes nobody writes about

1. KMS outage during the write path

This is the one that surprises teams. You designed the read path to tolerate KMS hiccups via cache. The write path cannot be served from cache — a new generation needs a fresh wrap, and even the steady-state write uses an active-write DEK that may not be in the local cache yet on a new pod.

When KMS goes down (regional outage, IAM misconfiguration, account-level throttle), writes start failing while reads keep working. Customers panic. Support escalates. Your dashboard looks fine because read traffic is healthy.

Three defenses, in order of how much we'd insist on each:

Pre-warm the cache on pod startup. When a pod boots, eagerly unwrap the active-write DEK for every tenant it might serve. Reject readiness until this completes. Pods that can't reach KMS at boot never enter rotation. This is the single highest-leverage change.
Allow a write_unavailable mode that fails fast and explicitly. Writers see SQLSTATE 53300 ("KMS unreachable") instead of timing out. Applications can queue, drop to a dead-letter, or retry with backoff — but they know what's happening, instead of staring at 30-second timeouts.
Cross-region KMS replication. AWS multi-region keys, GCP multi-region key rings. Adds cost; removes the regional-outage failure case. Required for any customer that's run a regional failover drill.

The defense we don't recommend: a synthetic "emergency unwrapped DEK" fallback. It defeats the whole point of BYOK. Two customers asked for it; both withdrew the request after their security team read the runbook.

2. Key rotation in a multi-region setup

You read the rotation post and built a clean two-stage rotation. Now you have three regions, async replication between them, and BYOK. The rotation now has a subtle ordering bug.

The flow that breaks:

t=0   region A: promote generation 7 to active-write
t=1   region A: a row is written, encrypted under DEK_7
t=2   replication ships the row to region B
t=3   region B receives the row, tries to decrypt → fails

      because region B hasn't yet seen the keyring update
      that says "generation 7 exists and here's its wrapped DEK"

The fix is order-of-operations: the keyring entry for a new generation must replicate before any row written under that generation can replicate. RedDB handles this internally by treating keyring updates as a special replication record that flushes ahead of data records carrying that generation. If you're integrating at a lower layer (e.g., custom replication), preserve this invariant or accept that cross-region reads will occasionally return decryption errors for the few seconds after a rotation.

The corollary: rotate during a quiet window the first time you do it on a new multi-region deployment, and watch the replication-lag-versus-keyring-lag metric. We expose both. They should never be reordered, but the time you find out they were is also the time you find out it matters.

3. The CMK is deleted (or scheduled for deletion)

Customer's engineer fat-fingers a aws kms schedule-key-deletion on the wrong CMK. The customer panics. They cancel the deletion within the 7-day window (AWS minimum). Crisis averted.

Or: they didn't notice for 8 days, and now the CMK is gone. Every row encrypted under DEKs wrapped by that CMK is unreadable. Forever. Including from your backups.

This is the actual, named threat that BYOK introduces, and it's the customer's responsibility — but you carry the operational scar of being the one who has to tell them. Three things to put in your runbook:

Monitor for kms:ScheduleKeyDeletion events on customer CMKs. Most customers will grant you the CloudTrail read permission for this; the ones that won't, you note in their onboarding doc.
Page on any read that returns KMSAccessDenied or KMSDisabled above a threshold rate. This is the signal of a CMK being yanked.
Document that backups encrypted under a yanked CMK are unrecoverable. This sounds obvious. It is not obvious to everyone. Get sign-off in writing during onboarding.

HSM-backed: same shape, more latency

CloudHSM, GCP EKM, on-prem HSMs (Thales, Entrust) plug in at the KMS layer — either as the KMS provider (rare), or behind a CMK that's configured to call into the HSM for every operation (common). From RedDB's side it looks like a slow KMS. Three operational deltas:

Latency is ~10–50ms per unwrap, vs ~5ms for cloud KMS. Cache TTL matters more. We've seen customers set dek_cache.ttl: 4h for HSM-backed setups; verify with their security team.
Throughput is capped by the HSM. A single CloudHSM cluster does ~1,200 RSA ops/s. Plan capacity for peak unwrap rate, which happens during a cache cold-start after a deploy.
Failover is the customer's problem. HSM clusters can lose quorum. Their runbook, not yours — but document the read/write impact when they do.

What to actually measure

If you operate BYOK in production, these five metrics catch 95% of the failure modes:

vault_kms_unwrap_seconds              (histogram, p50/p99 per tenant)
vault_kms_unwrap_errors_total         (counter, by reason: timeout, throttle, denied)
vault_dek_cache_hit_ratio             (gauge, per tenant)
vault_active_generations_count        (gauge, per tenant — should be 1 except during rotation)
vault_keyring_replication_lag_seconds (gauge, cross-region)

Page on:

vault_kms_unwrap_errors_total{reason="denied"} > 0 — almost always a yanked CMK.
vault_dek_cache_hit_ratio < 0.95 for 5 min — cache misconfigured or TTL too short.
vault_kms_unwrap_seconds{quantile="0.99"} > 500ms for 5 min — KMS is sick or HSM is overloaded.

The cache-hit-ratio alarm is the one that's caught the most weird configs in practice. It's cheap to wire up. Wire it up first.

The shape that ages well

BYOK is mostly boring once configured. The traps are the ones that show up months later when a customer rotates an IAM policy, deletes a CMK, fails over to a new region, or sells to a sub-organisation that wants its own CMK. The shape of the system that survives all of these:

Per-tenant CMKs from day one, even if you only have one tenant.
Cache TTL that matches the customer's revocation expectation, documented.
Pre-warming on pod startup, with readiness gated on it.
Cross-region keyring replication that flushes ahead of data.
Alerts on KMSAccessDenied and on cache-hit-ratio drops.

None of these are interesting in a sales call. All of them are the difference between "BYOK works" and "BYOK works in production, for years, with no surprises."

Per-row encryption with zero-downtime key rotation — the rotation mechanics this post assumes.
GDPR right-to-erasure when data is replicated and embedded — the cryptographic-deletion shape, fleshed out.
Self-hosting RedDB: the real-world checklist — where the KMS section in your runbook should live.

MongoDB → RedDB: the queries that change shape

2026-04-30

A MongoDB-to-RedDB migration is not the war story people expect. The replication is straightforward, the dual-write window is small, the cutover is boring. What is interesting is what happens to your queries — most of them get shorter, a few get longer, and three patterns change shape entirely. This post is a translation table for the ones that change shape.

We are not going to argue MongoDB versus relational here. We are going to assume you have already decided to move and want to know what your aggregation pipeline looks like on the other side. Every translation below has been pulled from a real migration we have shipped — names changed, shapes preserved.

The mental model shift

Before any single query: in Mongo you push computation up into the pipeline, and the document is the unit of co-location. In RedDB you push computation down into the query planner, and the row (with its JSONB columns) is the unit of co-location. The good news is that RedDB's JSONB support is rich enough that you do not have to flatten everything — keep the long tail of optional fields as JSONB and lift only the columns you query or index on.

The rule of thumb we use: if you filter, sort, or join on a field in more than one place, promote it to a column. Otherwise leave it in JSONB. Migrations that try to fully normalise every Mongo document on day one tend to stall around week four. Migrations that promote the hot fields and leave the long tail in JSONB tend to ship in a fortnight.

Translation 1: simple find with projection

The cheapest case. Mongo:

db.orders.find(
  { customer_id: "c_42", status: "open" },
  { _id: 1, total: 1, created_at: 1 }
).sort({ created_at: -1 }).limit(20)

RedDB:

SELECT id, total, created_at
FROM orders
WHERE customer_id = 'c_42' AND status = 'open'
ORDER BY created_at DESC
LIMIT 20;

Index on (customer_id, status, created_at DESC) and you are done. If status is one of a small set, a partial index WHERE status = 'open' is often cheaper. Nothing exotic.

Translation 2: aggregation pipeline → SQL with JSONB

This is where most of the work lives. A typical revenue rollup in Mongo:

db.orders.aggregate([
  { $match: { created_at: { $gte: ISODate("2026-01-01") }, status: "paid" } },
  { $unwind: "$items" },
  { $group: {
      _id: { sku: "$items.sku", month: { $dateToString: { format: "%Y-%m", date: "$created_at" } } },
      revenue: { $sum: { $multiply: ["$items.qty", "$items.unit_price"] } },
      orders:  { $addToSet: "$_id" }
  }},
  { $project: {
      sku: "$_id.sku",
      month: "$_id.month",
      revenue: 1,
      order_count: { $size: "$orders" },
      _id: 0
  }},
  { $sort: { month: 1, revenue: -1 } }
])

The RedDB version uses jsonb_array_elements for the unwind and ordinary GROUP BY:

SELECT
  item->>'sku'                                   AS sku,
  to_char(created_at, 'YYYY-MM')                 AS month,
  SUM((item->>'qty')::numeric * (item->>'unit_price')::numeric) AS revenue,
  COUNT(DISTINCT id)                             AS order_count
FROM orders, jsonb_array_elements(items) AS item
WHERE created_at >= '2026-01-01' AND status = 'paid'
GROUP BY sku, month
ORDER BY month, revenue DESC;

Three things to notice:

$unwind is jsonb_array_elements in a FROM clause. It produces one row per array element, which is exactly what the SQL aggregation expects.
$addToSet + $size is COUNT(DISTINCT) — shorter and faster, because the planner does not have to materialise the set.
$dateToString is to_char. If you do this rollup often, store to_char(created_at, 'YYYY-MM') as a generated column and index it.

For pipelines deeper than three stages we usually rewrite as a CTE chain, one CTE per $match/$group stage. The pipeline shape survives — only the syntax changes.

Translation 3: `$lookup` → `JOIN`

$lookup is a join with extra steps. The literal translation almost always works and is faster than the Mongo version.

Mongo:

db.orders.aggregate([
  { $match: { status: "open" } },
  { $lookup: {
      from: "customers",
      localField: "customer_id",
      foreignField: "_id",
      as: "customer"
  }},
  { $unwind: "$customer" },
  { $project: { _id: 1, total: 1, "customer.email": 1, "customer.tier": 1 } }
])

RedDB:

SELECT o.id, o.total, c.email, c.tier
FROM orders o
JOIN customers c ON c.id = o.customer_id
WHERE o.status = 'open';

The Mongo $lookup requires an index on customers._id (free) and traverses one document at a time. The SQL version is hash-joined or merge-joined in one pass. We have not yet seen a $lookup whose SQL equivalent was slower.

Translation 4: change streams → logical replication or triggers

This is one of the three shape-changers.

A Mongo change stream is a tailing cursor on the oplog:

const stream = db.orders.watch([{ $match: { "fullDocument.status": "paid" } }])
for await (const change of stream) {
  await fulfillment.enqueue(change.fullDocument)
}

RedDB offers two replacements, and they are not interchangeable:

Option A: logical replication (use this if you have one or two well-defined consumers that need every change, in order, durably):

CREATE PUBLICATION orders_pub FOR TABLE orders WHERE (status = 'paid');
-- consumer connects via the replication protocol

Option B: a trigger that writes to an outbox table (use this if you want application-level retry control or fan-out to a queue):

CREATE TABLE outbox_orders (
  id          bigserial PRIMARY KEY,
  order_id    text NOT NULL,
  payload     jsonb NOT NULL,
  created_at  timestamptz NOT NULL DEFAULT now(),
  consumed_at timestamptz
);

CREATE FUNCTION orders_to_outbox() RETURNS trigger AS $$
BEGIN
  IF NEW.status = 'paid' AND (OLD IS NULL OR OLD.status <> 'paid') THEN
    INSERT INTO outbox_orders (order_id, payload)
    VALUES (NEW.id, to_jsonb(NEW));
  END IF;
  RETURN NEW;
END;
$$ LANGUAGE plpgsql;

CREATE TRIGGER orders_outbox_t
AFTER INSERT OR UPDATE ON orders
FOR EACH ROW EXECUTE FUNCTION orders_to_outbox();

Your fulfillment worker pulls from outbox_orders with FOR UPDATE SKIP LOCKED, processes, then sets consumed_at. The outbox pattern gives you exactly-once-with-effort semantics that watch() did not have — change streams resume on a token, but the resume token is yours to lose, and we have watched several teams discover this the hard way.

The shape change to internalise: change streams are a transport; the outbox-plus-poller pattern is a storage layer with a transport on top. The latter is more code and more correctness.

Translation 5: GridFS → blob columns (or object storage)

GridFS is a workaround for Mongo's 16MB document limit. RedDB does not have that limit, but you still should not put a 50MB PDF in a row.

The translation depends on file size:

Mongo (GridFS)	RedDB	Use when
Files < 1 MiB	`BYTEA` column on the owning row	Small avatars, signatures
Files 1–8 MiB	`BYTEA` column with `STORAGE EXTERNAL`	Documents, medium attachments
Files > 8 MiB	S3-compatible object store, URL + sha256 row	Videos, large datasets

A typical migration moves the GridFS chunks collection straight to object storage, writes the metadata row to RedDB with a storage_url and sha256, and leaves a one-shot job that lazily pulls each blob the first time it is requested. The big-file branch is the right default — GridFS performance gets pathological once you have a few hundred thousand large files in one collection, and most teams quietly already keep them in S3 anyway.

Translation 6: text search

If you used Mongo's $text operator, you have one of two situations:

You needed light keyword search on a small set of fields. Use RedDB's built-in tsvector:

ALTER TABLE products ADD COLUMN search_v tsvector
  GENERATED ALWAYS AS (to_tsvector('english', name || ' ' || coalesce(description, ''))) STORED;
CREATE INDEX products_search_v_idx ON products USING gin (search_v);

SELECT id, name FROM products
WHERE search_v @@ plainto_tsquery('english', 'wireless headphones');

You needed real relevance, faceting, or typo-tolerance. Mongo $text was already not enough. Move to a real search index (Tantivy on RedDB, or an external Elastic/Meili) and pipe updates through the same outbox table from translation 4.

Translation 7: TTL indexes → scheduled DELETE

Mongo TTL indexes are a background job dressed as an index. RedDB ships the job explicitly:

CREATE INDEX sessions_expires_at_idx ON sessions (expires_at)
  WHERE expires_at IS NOT NULL;

-- run from cron, every minute
DELETE FROM sessions WHERE expires_at < now() LIMIT 10000;

The LIMIT matters. The Mongo TTL worker batches; your cron job should too, or a single sweep can lock half the table during peak.

What we left out and why

Two Mongo patterns do not get a translation here because they should not survive the migration:

Schemaless writes from many services. If five services write to the same collection with different shapes, the schemaless freedom was already costing you. The migration is the right moment to define the shape. Promote the shared fields to columns; let the disagreement live in JSONB only where it is genuinely unresolved.
Embedded arrays of unbounded growth. A comments: [...] field that grows for the lifetime of a post is a future operational problem in any database. The migration is the right moment to lift it into its own table.

The honest summary

Most query translations are mechanical and the result is faster, more inspectable code. The three shape-changers — change streams, GridFS, and unbounded embedded arrays — deserve real design time and should not be translated literally. Plan one to two days per shape-changer; the rest of the queries are an afternoon's typing.

If you are deep in a migration and stuck on a pipeline you cannot translate cleanly, send us the pipeline. We have probably seen its shape before.

Disaster recovery for RedDB: PITR, snapshots, regional failover

2026-04-29

This is the post we hand to operators after the self-host checklist. The checklist gets you running. This one gets you back up when something goes wrong — and tells you, with numbers, how long that takes.

DR posts tend to oversell. We won't. RedDB gives you good RPO and decent RTO on a single region, and asynchronous cross-region replication that is honest about its lag. It does not give you synchronous multi-region writes. If you need RPO=0 across continents, RedDB is the wrong answer today, and the non-applicability post says so out loud.

The three failures you plan for

Most operator panic falls into one of three shapes, and your DR strategy has to cover each one differently:

Logical corruption — a bad migration, an UPDATE without a WHERE, a poisoned write from a regression. You want to rewind to just before the bad thing happened. PITR is the answer.
Volume loss — the underlying disk is gone (rare on managed storage, possible on bare metal). The cluster is otherwise healthy elsewhere. Snapshot + WAL replay is the answer.
Region loss — the whole AZ or region is gone or unreachable for hours. Cross-region replica promotion is the answer.

These three call for different artifacts (continuous WAL archive, periodic full snapshot, follower replica) and different RPO/RTO targets. Sizing all three correctly is most of the work.

Point-in-time restore (logical corruption)

Logical corruption is the most common DR scenario in practice. It is also the one where naive "we have backups" answers fail — a nightly snapshot does not help if the bad write happened at 14:30 and the next snapshot is at 02:00.

RedDB's PITR machinery is two pieces:

Base snapshot — a consistent full copy taken on a schedule (default: daily at 02:00 UTC, before the EU traffic ramp).
Continuous WAL archive — every WAL segment is shipped to object storage as it seals, typically every 16 MiB or every 30 seconds (whichever comes first).

To restore to an arbitrary point in time, you fetch the most recent snapshot before the target time, then replay the archived WAL forward up to the requested timestamp.

# Restore to 2026-04-28 14:29:00 UTC (one minute before the bad UPDATE)
reddbctl restore \
  --target-time '2026-04-28T14:29:00Z' \
  --snapshot-bucket s3://reddb-backups-prod \
  --wal-bucket s3://reddb-wal-prod \
  --destination /var/lib/reddb-restore \
  --master-key-arn arn:aws:kms:us-east-1:1234:key/abcd-ef

The command fetches the snapshot taken at 02:00 the same day (~12 hours of WAL to replay), unwraps the KMS-encrypted at-rest key, applies the WAL segments in order, and stops at the timestamp you asked for.

In practice you do not restore in place over a live cluster. You restore to a parallel data directory, point a single read-only RedDB process at it, run your forensic queries to confirm the rewind landed on the right side of the bad write, then decide whether to swap the live cluster or surgically extract rows back into it. The second option (surgical extraction) is almost always the right one — full-cluster rewind discards every legitimate write between the bad event and now.

RPO for PITR: bounded by your WAL ship interval. Default is 30 seconds, meaning the worst case is 30 seconds of writes lost if the region itself dies between WAL ships. For an in-region logical corruption restore (the common case), RPO is effectively zero — the WAL has already been shipped before you decide to restore.

RTO for PITR: bounded by snapshot size and WAL replay rate. On a 200 GiB dataset with one day of WAL to replay, expect 20–40 minutes wall-clock. Most of that is the snapshot download, not the replay.

Snapshot scheduling

The default daily-at-02:00 snapshot is fine for most teams. The two cases where you change it:

You have a strict RTO and 12+ hours of WAL replay would blow it. Snapshot twice a day, or every 6 hours. Cost scales linearly with snapshot frequency — full snapshots are expensive on large datasets.
Your write pattern is bursty and a 24-hour replay would include the daily batch job that writes 100 GiB. Snapshot after the batch job, not before.

The G-27 checklist post showed the CronJob manifest. Here is the schedule decision:

# values.yaml — snapshot schedule decision points
snapshots:
  schedule: "0 2 * * *"              # daily 02:00 UTC
  retention_days: 35                  # > 30-day audit window + buffer
  encryption_key_arn: ${KMS_KEY_ARN}  # always present, never optional
  destination: s3://reddb-backups-prod
  # If your RTO is < 4 hours, uncomment and run twice daily:
  # schedule: "0 2,14 * * *"

Two retention rules:

Snapshots: keep ≥ (your max RTO window) + 7 days of full snapshots. The +7 covers "we discovered the corruption two days late."
WAL: keep ≥ 35 days of WAL archive even though that's longer than most snapshot retention. The reason: an auditor or legal hold sometimes asks for "data as it existed on date X," and WAL replay is the only way to answer that without keeping every snapshot forever.

Both buckets must be in a different failure domain than the live cluster — different region or, at minimum, a different storage account with separate IAM. The most common DR mistake we see is backups stored in the same bucket as the cluster's working data. When the cluster's bucket dies, so do the backups.

Cross-region replication topology

RedDB ships an asynchronous replication mode where one or more follower clusters in other regions stream the WAL from the leader and apply it. This gives you:

A warm-standby cluster you can promote if the leader region dies.
A read-only replica in another region for read-locality (a useful side benefit).
A continuously-tested DR target — if replication is healthy, you know the recovery path works right now, not "in theory next quarter."

The topology we recommend for most teams is one leader and one async follower, both behind their own load balancer, with DNS pointed at the leader by default.

┌─────────────────┐         WAL stream         ┌─────────────────┐
│ Leader (us-east)│ ────────────────────────▶  │ Follower (eu-w) │
│   r/w traffic   │                            │   read-only     │
└────────┬────────┘                            └────────┬────────┘
         │                                              │
         ▼                                              ▼
   app traffic                                  read-only fan-out
       (default)                              (optional, lower-priority)

The replication lag metric (replication_lag_bytes and replication_lag_seconds) is one of the four metrics the G-27 checklist said to alert on. For most workloads, expect 0.5–3 seconds of lag, with bursts to ~30 seconds during compaction. If you see sustained lag > 60 seconds, page someone — the follower is no longer a viable failover target until it catches up.

Promoting a follower

When the leader region is gone (or sufficiently broken that you've made the call to fail over), promote the follower:

# On the follower cluster:
reddbctl promote \
  --confirm-leader-down \
  --new-leader-region eu-west-1

The --confirm-leader-down flag is intentional friction. We have seen exactly one customer accidentally promote a follower while the leader was still serving writes; the resulting split-brain took six hours to clean up. The flag forces an operator to acknowledge they have already verified the leader is unreachable from the application tier and from the follower itself.

After promotion, your DNS or service-mesh routing has to swap. We do not automate the DNS swap. Most teams' DNS automation is bespoke and brittle, and an automatic regional failover that fires on a transient network blip is worse than the disaster it was supposed to mitigate. Manual swap, executed by a human who has read a runbook, is the right tradeoff for almost everyone.

RPO for regional failover: equal to replication lag at the moment of leader loss. If lag was 2 seconds, you lose 2 seconds of writes. If lag was 60 seconds, you lose 60 seconds.

RTO for regional failover: dominated by DNS propagation + connection-pool reset in the application tier. Cluster-side promotion is sub-30s. Realistic end-to-end RTO is 5–15 minutes depending on how aggressive your DNS TTLs are and whether your application has connection retries that survive a backend swap.

The numbers you can hit

For a midsize deployment (200 GiB dataset, 5K writes/sec, daily snapshot, 30s WAL ship interval, one async cross-region follower):

Failure	RPO	RTO	Tooling
Logical corruption	~0 (in-region)	20–40 min	`reddbctl restore --target-time`
Volume loss	< 30 sec	15–25 min	Snapshot + WAL replay onto new PV
Region loss	2–30 sec	5–15 min	`reddbctl promote` + DNS swap
Whole-account loss	1 day	4+ hours	Restore from cross-account vault

These are not aspirations. They are the numbers we measure on our own deployments. Your numbers will differ — bigger datasets push RTO up roughly linearly, faster networks pull it down, and applications with aggressive connection retries shave 1–2 minutes off the failover number.

If your business needs better than these, the answer is not to tune RedDB harder. The honest answer is below.

What we don't yet do

DR sales decks lie. Here is what RedDB explicitly does not provide today, and what you should pick instead if you need it:

Synchronous multi-region writes — no. Writes commit on the leader before being replicated. If you need RPO=0 across regions, look at Spanner, CockroachDB, or YugabyteDB. They pay a meaningful latency cost on the write path; that cost is unavoidable for the property you're buying.
Automatic regional failover — no. The promote command is manual on purpose. See above.
Cross-region read-after-write consistency — no. The follower is eventually consistent. If your application reads through the EU follower and expects writes from the US leader to be visible immediately, you will see stale reads. Pin reads to the leader for that workload, or accept the lag.
In-cluster geo-distribution of a single dataset — no. Each cluster is regional. The follower is a replica, not a shard. If you need data residency by tenant (some EU customers, some US), run separate clusters per region and route by tenant ID at the application layer.
Backup verification automation — partial. We ship a reddbctl restore --dry-run that verifies snapshot and WAL integrity without writing to disk. We do not automatically restore your nightly snapshot into a scratch cluster every day. That drill remains a human responsibility, and it is item #8 on the self-host checklist for a reason.

This list will shrink. The order matters: synchronous multi-region writes are not on the roadmap (the latency cost is not what most of our users want); cross-region read-after-write is on the roadmap (via causal tokens) and is the closest item to landing.

The drill you owe yourself

Pick a Tuesday. Set a timer. Pretend the leader region is gone.

Page yourself the way the real alert would page you.
Read your own runbook end-to-end. (You have a runbook, right?)
Promote the follower.
Swap DNS.
Verify the application is serving from the new region.
Read at least one row that was written less than 60 seconds before "the leader died." Verify it's there. Note what was lost if anything.
Restore the leader region. Decide whether to fail back or stay on the new leader.

The first time a team runs this drill, it takes 90 minutes longer than they estimated, they discover three runbook errors, and the application's connection-pool retry logic is wrong in some non-obvious way. That's the entire point of running it before the real outage. Schedule the next one for 90 days out.

TL;DR

Three failure shapes (logical corruption, volume loss, region loss) need three different DR artifacts. Cover all three or you don't have DR; you have hope.
PITR via base snapshot + continuous WAL archive: RPO ~zero for in-region restores, RTO 20–40 minutes on a midsize dataset.
Async cross-region follower: RPO 2–30 seconds, RTO 5–15 minutes including DNS swap.
Promotion is manual on purpose. Automatic regional failover causes more outages than it prevents.
No synchronous multi-region writes. If you need RPO=0 across continents, pick a different database.
Run the drill quarterly. The first drill always finds three real bugs.

Out of Firebase, into a real backend

2026-04-28

Firebase is a fantastic product for the first six months of a startup and an expensive constraint somewhere around month eighteen. This post is for the teams who are past month eighteen.

We have done this migration four times for customers in the last year. Same shape every time: founders love Firebase, the senior engineer who joined to "harden the backend" hates it, and somewhere in between is a real product with paying users that cannot afford a multi-week outage. The walkthrough below is the playbook we hand to the senior engineer on day one. It is not "Firebase bad, please switch" — it is "here is the order of operations, here is what is genuinely worse on the other side, and here is how you keep the lights on while you cut over."

Why teams leave (be honest about this first)

Before any migration, name the pain. We see four recurring drivers, roughly in order of severity:

Cost cliff at scale. Firestore reads at $0.06 per 100k start to dominate the bill once you have a real timeline or feed product. The pricing model penalises the exact access pattern (fan-out reads) that makes Firebase pleasant to build on.
Query expressiveness ceiling. Firestore has no joins, no aggregations beyond count, and composite indexes you have to predeclare. The third time someone says "we'll just denormalise it" is the moment you have built a relational schema in a non-relational database without any of the relational tools.
Vendor lock-in starts to hurt. Security rules, Functions, and the Firebase SDK leak deeply into the app. Migrating later is a bigger project than migrating now. Senior engineers who have seen this movie before push for the cut earlier rather than later.
Observability gap. Firebase Console is wonderful for the first 100 documents and useless at 100 million. You cannot EXPLAIN ANALYZE a Firestore query. You cannot trace why a single document read is slow.

If your pain is something else — say, you simply do not like Google as a vendor — be careful. Migrations are expensive. Spend the migration budget where the pain actually is.

The four bridges you are about to build

A Firebase-to-RedDB migration is four parallel bridges that you cut over independently:

Bridge	Firebase side	RedDB side
Auth	Firebase Auth tokens	RedDB sessions (or your own JWT)
Documents	Firestore collections	RedDB tables with JSON columns
Authorization	Security rules	Row-level policies
Real-time	`onSnapshot` listeners	Change streams over websocket

Build all four. Cut over auth first (it is the foundation), then run the document layer in dual-write, then move security to the server, then finally swap real-time. The order matters because each bridge depends on the previous one being stable.

Bridge 1: auth

Firebase Auth gives you a signed JWT per user with claims like sub (Firebase UID) and email. RedDB does not care where the JWT came from as long as you can verify it. The migration is therefore a verification gateway: trust Firebase tokens during the cutover, and issue your own RedDB session in exchange.

// During cutover, your /auth/bridge endpoint accepts a Firebase ID token,
// verifies it against Google's public keys, and returns a RedDB session.

import { verifyIdToken } from 'firebase-admin/auth'

export async function bridgeFirebaseToken(idToken: string) {
  const decoded = await verifyIdToken(idToken)
  // decoded.uid is the Firebase UID — keep it on the user row for lookup.

  let user = await reddb.query(
    'select id from users where firebase_uid = $1',
    [decoded.uid],
  ).first()

  if (!user) {
    user = await reddb.query(
      `insert into users (firebase_uid, email, created_at)
       values ($1, $2, now())
       returning id`,
      [decoded.uid, decoded.email],
    ).first()
  }

  return issueSession(user.id) // your own opaque session token
}

Two things to notice. First, firebase_uid becomes a permanent column on users — keep it forever; it is your join key for any future Firebase data you discover. Second, the endpoint creates the RedDB user row lazily on first sign-in. This means the cutover does not require a "migrate every user up front" batch job — users arrive as they sign in.

After three to four weeks, every active user has a RedDB session and you can issue your own tokens directly. Firebase Auth becomes a fallback for the long tail.

Bridge 2: documents

This is the largest bridge. Firestore's collection-document model maps cleanly to a table with one JSON column, but the lazy translation is a trap. Naive port:

create table firestore_documents (
  collection text not null,
  doc_id     text not null,
  data       jsonb not null,
  updated_at timestamptz not null default now(),
  primary key (collection, doc_id)
);

This works for week one and then you realise every query is a full table scan or a hand-built JSONB GIN index per access pattern. Do not do this.

Instead, model each Firestore collection as a real RedDB table. Pull the fields you actually query into typed columns and keep an extra JSONB column for the long tail. Example: a Firestore users/{uid} document with { email, displayName, plan, preferences: {...} } becomes:

create table users (
  id           uuid primary key default gen_random_uuid(),
  firebase_uid text unique,
  email        text not null,
  display_name text,
  plan         text not null default 'free',
  preferences  jsonb not null default '{}'::jsonb,
  created_at   timestamptz not null default now(),
  updated_at   timestamptz not null default now()
);

create index users_plan_idx on users (plan);

The plan column gets the index because you filter on it. preferences stays in JSONB because no one queries inside it — it is read whole and written whole.

Dual-write during cutover

You cannot atomically migrate a live document store. Instead, run a window of dual-write: every write goes to both Firestore and RedDB, every read is configurable per surface. The pattern that survives production looks like this:

async function updateUserPlan(uid: string, plan: string) {
  // 1. Write to RedDB first (it is the new source of truth).
  await reddb.query(
    `update users set plan = $1, updated_at = now()
     where firebase_uid = $2`,
    [plan, uid],
  )

  // 2. Best-effort mirror to Firestore. If it fails, log — do not roll back.
  try {
    await firestore.collection('users').doc(uid).update({ plan })
  } catch (err) {
    logger.warn({ err, uid }, 'firestore mirror failed — already on reddb')
  }
}

RedDB is the new source of truth from day one of cutover. Firestore is a write-behind cache for any old client code you have not yet migrated. The asymmetry is deliberate: you can read stale data from Firestore for a week, but you cannot lose data that is now only on RedDB.

The backfill job

For documents that existed before the cutover, a one-shot backfill copies them across. Use SKIP LOCKED so multiple workers can run in parallel without stepping on each other:

with batch as (
  select id from firebase_users_to_migrate
  where migrated_at is null
  order by id
  limit 500
  for update skip locked
)
update firebase_users_to_migrate m
set migrated_at = now()
from batch
where m.id = batch.id
returning m.id, m.firestore_doc;

The worker reads firestore_doc (a JSON snapshot loaded from Firestore), translates it to typed columns, inserts into users, and marks the row migrated. Idempotent on firebase_uid uniqueness — safe to re-run.

Bridge 3: authorization (security rules → row-level policies)

Firestore security rules are a thicket of match /users/{uid} { allow read: if request.auth.uid == uid } declarations that mix authentication, authorization, and validation into a single mini-DSL. The good news: every rule you actually use translates to one of three RedDB patterns.

Pattern A: owner-only access.

create policy users_self on users
  using (id = current_setting('app.user_id')::uuid);

The application sets app.user_id at the start of each request based on the verified session. Every query against users is implicitly filtered.

Pattern B: membership-based access.

create policy team_member on documents
  using (
    exists (
      select 1 from team_members
      where team_members.team_id = documents.team_id
        and team_members.user_id = current_setting('app.user_id')::uuid
    )
  );

Pattern C: role-based access.

create policy admin_all on audit_log
  for select
  using (current_setting('app.user_role') = 'admin');

Most Firestore rules are some combination of these three. The translation is mechanical; the win is that policies live in the database and apply to every query path — including the ad-hoc psql session you open at 2am, which Firestore could never protect.

Bridge 4: real-time

This is the bridge where you lose the most magic, and we will not pretend otherwise.

Firebase onSnapshot is genuinely brilliant. You attach a listener to a query and your UI updates whenever any document in the result set changes — with no server you have to write, no websocket you have to manage, no reconnection logic. It is the killer feature.

The RedDB equivalent is a change-stream subscription over websocket. You write the websocket server (or use ours) and clients subscribe to a query. The UX is the same; the implementation is now your problem.

// Server: stream changes for a user's documents.
reddb.subscribe('user_documents', {
  where: 'user_id = $1',
  params: [userId],
}).on('change', (event) => {
  ws.send(JSON.stringify(event))
})

What gets worse: you now own connection management, reconnection, backpressure, and the "what happens when the client missed 30 seconds of updates" question. Firebase handled all of this transparently. Plan for two engineer-weeks on this bridge alone if you do not already have a websocket layer.

What gets better: the change stream is a real database stream. You can filter, transform, batch, and join it with other tables. You can replay from any LSN. You can hook into it for analytics, audit, or webhooks without paying Firestore for a second read of every document.

The things that are genuinely worse

Honest reckoning. Here is what you give up:

Firebase Console. There is no equivalent admin UI on day one. You will write queries in psql or build a small internal admin tool. Plan a sprint for this; do not pretend the existing Postgres GUIs are a substitute for non-technical staff.
Offline-first mobile SDK. Firestore caches and syncs offline. RedDB does not. If you have an offline-heavy mobile app, you are building a sync layer yourself — this might be a reason to delay the migration until you have the staffing for it.
Magic scaling for write-heavy fan-out. Firestore happily eats a million-write fan-out and bills you for it. RedDB will too, but you will see the load on your own servers and have to provision for it. The bill is more visible (which is the point) but so is the operational responsibility.
Drop-in mobile auth. Firebase Auth's SDK handles password reset, social login, MFA, and email verification out of the box. You can replicate all of this on RedDB — but it is now code you maintain, not a checkbox in the console.

If you are honest about the four above and your team still wants to migrate, you are ready.

A four-week cutover plan

Week 1: Stand up RedDB, build the auth bridge, run dual auth (Firebase Auth verifies, RedDB session issued). Zero user-visible changes.

Week 2: Build the document tables for your top-three highest-traffic collections. Start dual-write. Backfill historical data. Reads still come from Firestore.

Week 3: Flip reads to RedDB for those three collections. Monitor for one week. Migrate the remaining collections in priority order.

Week 4: Translate security rules to row-level policies. Cut over real-time listeners. Begin decommissioning Firestore reads. Keep Firestore as a write-behind mirror for two more weeks as insurance.

Past four weeks: turn off the Firestore mirror, cancel the contract, write the postmortem.

TL;DR

Migration is four bridges (auth, documents, authorization, real-time) cut over in order. Dual-write keeps you safe through the cutover; RedDB is the source of truth from day one. You lose Firebase Console, offline-first sync, and SDK magic — be honest about which of those actually matter for your product before you start. Plan four weeks, expect six, and budget one full engineer for the websocket layer if real-time matters to you.

The companion piece in this series (When NOT to migrate: a checklist) is just as important. If your pain is not on the list at the top of this post, the right answer might be to stay on Firebase for another year. Migrations are expensive — spend the budget where the pain actually is.

The drift window: why your RAG retrieves stale chunks

2026-04-27

A support ticket landed on a Tuesday. A customer's internal assistant was telling employees the wrong on-call rotation. The runbook in the source-of-truth wiki was correct. The bot's answer was confidently, specifically wrong — naming an engineer who had rotated off two weeks earlier.

The wiki page had been edited eight times since that engineer rotated off. The retrieval still pulled the chunk from before the first edit.

This is the canonical shape of the drift window bug. The anchor post for this pillar (RAG without a second database) defines the window. This post is about how it actually breaks production, why your monitoring probably won't catch it, and what specifically you have to change in the write path to make it impossible.

What the customer saw

The on-call rotation page in the wiki had this history:

T-21d  Page created. On-call: Anita.
T-14d  Anita rotates off. Page edited: On-call: Brian.
T-12d  Brian's title gets fixed. Page edited.
T-09d  Holiday coverage added. Page edited.
T-02d  Phone numbers updated. Page edited.
T-00   Employee asks the bot: "who's on call this week?"
T-00   Bot answers: "Anita is on call."

Every edit since T-14d enqueued a re-embed job. Five jobs. The queue worker had been wedged for ten days behind a poison-pill row that nobody had alerted on. The vector store still held the embedding of the T-21d revision. The retrieval was precisely correct — the chunk it returned was the best match for the query, given what was in the index. The chunk was just twenty-one days out of date.

The bot did not hallucinate. The pipeline returned stale ground truth.

Why monitoring missed it

The team had monitoring. It looked roughly like this:

Queue depth alert: pages at >10k pending jobs.
Worker error rate: pages at >1% of jobs erroring.
Embedding API latency: pages on P99 >3s.
Vector store write latency: pages on P99 >500ms.

None of these fired. The queue depth was steady — five jobs sat behind the poison pill, and the rest of the system kept up. The worker wasn't erroring; it was waiting. The embedding API was fine. The vector store was fine.

What was missing is the one metric that actually corresponds to the bug: the time between a source row's updated_at and the time its embedding row was last written. Call it drift_age_seconds. Without that metric, a queue can be "healthy" by every other measure while individual rows are silently weeks stale.

Most monitoring stacks don't compute this because computing it requires joining the source store to the vector store, which usually live in different databases. So the metric gets postponed. Then the bug ships.

Why retries don't save you

The reflexive fix to a drift-window incident is to make the pipeline more robust:

Better retry logic on the worker.
Idempotency keys on embedding requests.
A separate "stale-row scanner" that periodically diffs source and vector store and re-enqueues.
A health-check job that picks a random row and verifies the embedding is current.

These help. They do not close the window. They shorten it, on average, while leaving the tail unchanged. A poison pill, a deploy gone sideways, a partial outage of the embedding provider — any of these and the window reopens for whichever rows get stuck behind the failure. The customer that hits one of those rows during the window sees the bug.

The window exists because the source-of-truth write and the embedding write are two writes to two systems. Any architecture that keeps that property has a drift window. Architectures that collapse it into one write do not.

The fix: one transaction, both columns

The anchor post walks through the schema. The short version: the embedding column lives on the same row as the source text. The write that updates the text is the write that updates the embedding. No queue, no worker, no separate store.

The wiki-page table looks like this:

CREATE TABLE wiki_pages (
  id            UUID PRIMARY KEY,
  slug          TEXT UNIQUE NOT NULL,
  body          TEXT NOT NULL,
  body_tsv      tsvector GENERATED ALWAYS AS (to_tsvector('english', body)) STORED,
  embedding     vector(1536),
  embed_model   TEXT,
  embedded_at   TIMESTAMPTZ,
  updated_at    TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE INDEX ON wiki_pages USING hnsw (embedding vector_cosine_ops);

The edit endpoint runs:

BEGIN;

  UPDATE wiki_pages
     SET body = $1,
         updated_at = now()
   WHERE slug = $2
  RETURNING id, body;

  -- application embeds $body inside the same request
  UPDATE wiki_pages
     SET embedding   = $3,
         embed_model = $4,
         embedded_at = now()
   WHERE id = $5
     AND updated_at = $6;  -- CAS guard: did anyone else win the race?

COMMIT;

The embedding call happens between the two UPDATE statements, inside the same transaction. The transaction commits only if both writes commit. The retrieval index sees body and embedding change atomically.

There is no queue. There is no worker. There is no second store. The drift window for this row is zero by construction.

What you give up

This is a tradeoff, not a free lunch. Three things change:

Write latency includes the embedding call. A wiki edit no longer returns in 30ms. It returns in 200-800ms, dominated by the embedding model. For an editor UI this is fine — the human just clicked save. For a hot write path (logging, telemetry, high-frequency events) it is unacceptable, and you do want a queue. The drift window is the price you pay for write-path simplicity; for high-frequency writes the price is wrong.

A failed embedding call fails the write. If the embedding provider is down, you cannot save the document. This is usually the wrong behavior. The pattern around it is to make the embedding nullable, let the write commit with embedding IS NULL, and have a backfill worker pick up the null rows. The retrieval query filters out null-embedding rows or falls back to text search for them. The window for those specific rows is non-zero, but it is bounded to "this row was just created" rather than "any row that's ever been edited."

Re-embedding the whole corpus is no longer a separate pipeline. Model upgrades require a migration, not just a worker config change. The anchor post covers the SKIP LOCKED pattern for this. It's straightforward but it is a thing you now have to write.

How to know if you have this bug right now

Three checks you can run today against your existing two-store RAG pipeline:

Pick ten source rows that were edited in the last month. For each, fetch the source updated_at and the vector-store's last-write timestamp for that row. Compute the gap. If any gap is more than a few minutes, you have a drift window in production right now.
Look at your worker's oldest pending job age. Not the queue depth — the age of the oldest job. If it's older than your error budget allows, your monitoring is alerting on the wrong dimension.
Search your incident history for the phrase "stale" or "outdated" in customer-reported bugs. Drift-window bugs almost always get reported as "the assistant gave me old information." They rarely get attributed to the pipeline because the pipeline looks healthy.

If any of these turn up something, the drift window is your bug. The fix is in the schema, not the queue.

TL;DR

The drift window is the gap between a source-row update and its embedding update.
Standard pipeline monitoring (queue depth, error rate, latency) does not detect it. The metric you need is per-row drift_age_seconds, which most stacks don't compute.
Retries and stale-row scanners shorten the window but cannot close it; the window exists because the source write and the embedding write are two writes to two systems.
Putting the embedding on the source row and updating both in one transaction collapses the window to zero, at the cost of write latency and an embedding-provider dependency on the write path.
For editor-driven content, the tradeoff is almost always worth it. For high-frequency writes, keep the queue and accept the bounded window.

Related: RAG without a second database (anchor), Building an agent memory layer on RedDB (atomic episodic+semantic writes use the same pattern).

Building an agent memory layer on RedDB

2026-04-26

Every agent project I've watched ship spends two weeks deciding what "memory" means before writing a line of retrieval code. This post tries to skip that fortnight. It is a concrete schema — three tables, four indices, and the queries that wire them together — that you can copy into your own RedDB-backed project and start tuning instead of designing.

The schema is opinionated. It distinguishes episodic memory (what happened, in order), semantic memory (facts the agent extracted from those events), and procedural memory (cached outcomes of expensive tool calls). The boundaries between them matter: they have different retention rules, different retrieval paths, and different invalidation triggers. Collapsing them into one big memories table is the most common mistake — and the one that turns a six-week agent project into a six-month one.

This post is the practical follow-up to RAG without a second database. The same single-database principle applies: the vector lives on the row it describes, the same transaction writes both, and retrieval is one query instead of a fan-out.

The three shapes of memory

Before the SQL, the distinctions.

Episodic memory is the raw stream of agent events: user turns, tool calls, tool results, model outputs, errors. It is append-only, ordered, and (mostly) immutable. You query it by conversation, by time range, and occasionally by similarity to find "have I seen this before." Retention is bounded — most episodic rows are useful for hours to days, not months.

Semantic memory is what the agent has extracted from episodic events: "the user's deploy target is us-east-1," "the user prefers concise responses," "the customer's plan is enterprise." These are durable facts. They are not append-only — facts get updated, contradicted, and superseded. Retrieval is by similarity (vector) and by tagged subject (the user, the project, the customer). Retention is long.

Procedural memory is cached outcomes of expensive or non-deterministic tool calls: the result of a web fetch, a search query, an LLM-generated summary of a long document. Procedural rows have a clear cache-key (the tool name plus its arguments) and an explicit expiry. Retrieval is by exact key — vectors are not the access path here, even though the content might be embedded for separate semantic retrieval.

Three shapes, three access patterns, three retention rules. One database.

The schema

-- Episodic: append-only stream of agent events.
CREATE TABLE episodic_events (
  id            BIGSERIAL PRIMARY KEY,
  session_id    UUID NOT NULL,
  ts            TIMESTAMPTZ NOT NULL DEFAULT now(),
  actor         TEXT NOT NULL,  -- 'user' | 'agent' | 'tool:<name>' | 'system'
  kind          TEXT NOT NULL,  -- 'message' | 'tool_call' | 'tool_result' | 'error'
  payload       JSONB NOT NULL,
  tokens_in     INTEGER,
  tokens_out    INTEGER,
  embedding     VECTOR(1536),   -- nullable; embed only when you'll retrieve
  embed_model   TEXT
);

CREATE INDEX episodic_session_ts ON episodic_events (session_id, ts DESC);
CREATE INDEX episodic_embed     ON episodic_events USING hnsw (embedding vector_cosine_ops)
  WHERE embedding IS NOT NULL;

-- Semantic: extracted facts the agent should remember.
CREATE TABLE semantic_facts (
  id            UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  subject       TEXT NOT NULL,  -- 'user:42' | 'project:acme' | 'global'
  predicate     TEXT NOT NULL,  -- short label: 'prefers', 'deploy_target', 'plan_tier'
  object        TEXT NOT NULL,  -- the fact value
  confidence    REAL NOT NULL DEFAULT 1.0,
  source_event  BIGINT REFERENCES episodic_events(id),
  embedding     VECTOR(1536) NOT NULL,
  embed_model   TEXT NOT NULL,
  created_at    TIMESTAMPTZ NOT NULL DEFAULT now(),
  superseded_by UUID REFERENCES semantic_facts(id),
  superseded_at TIMESTAMPTZ
);

CREATE INDEX semantic_subject ON semantic_facts (subject)
  WHERE superseded_at IS NULL;
CREATE INDEX semantic_embed   ON semantic_facts USING hnsw (embedding vector_cosine_ops)
  WHERE superseded_at IS NULL;

-- Procedural: cached tool outcomes.
CREATE TABLE procedural_cache (
  cache_key     TEXT PRIMARY KEY,           -- hash(tool_name || canonical(args))
  tool_name     TEXT NOT NULL,
  args          JSONB NOT NULL,
  result        JSONB NOT NULL,
  computed_at   TIMESTAMPTZ NOT NULL DEFAULT now(),
  expires_at    TIMESTAMPTZ NOT NULL,
  hit_count     INTEGER NOT NULL DEFAULT 0
);

CREATE INDEX procedural_expires ON procedural_cache (expires_at);

A few notes on the choices before we get to the queries.

The episodic embedding column is nullable. Most events — system messages, tool-call envelopes, error stack traces — will never be retrieved by similarity. Embedding them costs money and bloats the HNSW index for no benefit. Embed only the events you actually search: user turns and final agent responses, typically.

Semantic facts use soft-supersedes, not hard updates. When the user changes their preference, you don't UPDATE the row — you insert a new row and set the old row's superseded_by. The partial index WHERE superseded_at IS NULL keeps the live view small, and the full history is still there for "when did this change?" queries. This pattern matters more than people admit: the moment an agent contradicts itself with a stale fact, the contradiction is debuggable only if you can see the old fact and the event that should have superseded it.

Procedural cache keys are hashes of canonicalised args, not the raw JSON. Canonicalise means: sort keys, normalise whitespace, lowercase the tool name. Skip this and the same logical call with arguments in a different order becomes a cache miss, and your "cache" runs at a 20% hit rate while you wonder why the agent is slow.

Writing memory: the agent's tick

The hot path is the agent loop. Every tick, the agent appends to episodic, occasionally extracts a fact into semantic, and routinely reads-and-writes procedural. The constraint is that all of this must be transactional with respect to one event — partial writes mean the agent's view of its own past disagrees with its view of its own outputs.

-- Append a user turn and embed it in one transaction.
BEGIN;

WITH event AS (
  INSERT INTO episodic_events (session_id, actor, kind, payload, embedding, embed_model, tokens_in)
  VALUES ($1, 'user', 'message', $2::jsonb, $3::vector, 'text-embedding-3-small', $4)
  RETURNING id
)
INSERT INTO semantic_facts (subject, predicate, object, source_event, embedding, embed_model)
SELECT $5, $6, $7, event.id, $3::vector, 'text-embedding-3-small'
FROM event
WHERE $6 IS NOT NULL;  -- only insert the fact if extraction produced one

COMMIT;

One transaction, one embedding cost, one round-trip. The fact row points to the event that produced it; the event row carries the same embedding it would have written to a sibling vector store in the two-database version. The drift window from the RAG post is gone in the same way it was gone there: you cannot read the event without seeing the fact, because they were inserted under the same BEGIN.

Procedural writes are simpler — an INSERT ... ON CONFLICT (cache_key) DO UPDATE keyed on the canonicalised hash. The only subtlety is to also bump hit_count on every read (not just on the conflict path), which you do with a separate UPDATE on the cache row at retrieval time. Without that counter you have no signal for which cached tool calls are pulling weight and which are dead weight that should be evicted early.

Reading memory: the four queries

Four queries cover ~90% of agent retrieval needs.

1. Recent context. The last N events of the current session, in order. This is the conversation history you feed back to the model.

SELECT actor, kind, payload, ts
FROM episodic_events
WHERE session_id = $1
ORDER BY ts DESC
LIMIT $2;

The compound index (session_id, ts DESC) makes this a range scan. No vector cost, no fan-out — this is the cheapest query in the system and you will run it many times per tick.

2. Similar past events. Used for "have I seen something like this user question before, across sessions?" This is where episodic embeddings earn their keep.

SELECT id, session_id, ts, payload,
       1 - (embedding <=> $1::vector) AS similarity
FROM episodic_events
WHERE embedding IS NOT NULL
  AND ts > now() - interval '30 days'  -- bound the search by retention
ORDER BY embedding <=> $1::vector
LIMIT 10;

The ts > now() - interval '30 days' clause is the retention boundary doing real work — without it, the HNSW search degrades over time and the results get less and less relevant because older events dominate.

3. Subject's live facts. Everything the agent currently believes about a user or project.

SELECT predicate, object, confidence, created_at
FROM semantic_facts
WHERE subject = $1
  AND superseded_at IS NULL
ORDER BY confidence DESC, created_at DESC;

The partial index WHERE superseded_at IS NULL means this is an index-only scan in the common case. Live facts only. Supersession bookkeeping pays off here: you never accidentally feed the model a stale belief alongside the current one.

4. Cache lookup. The procedural read-and-bump.

UPDATE procedural_cache
SET hit_count = hit_count + 1
WHERE cache_key = $1
  AND expires_at > now()
RETURNING result, computed_at;

The UPDATE ... RETURNING pattern collapses read + write-back-stats into one round-trip. If the row is missing or expired, the result set is empty and the agent falls through to recomputing.

Supersession in practice

The semantic-supersession pattern needs one helper. When the agent decides a new fact contradicts an existing one, mark the old one superseded in the same transaction as the new insert:

BEGIN;

INSERT INTO semantic_facts (subject, predicate, object, source_event, embedding, embed_model, confidence)
VALUES ($1, $2, $3, $4, $5, $6, $7)
RETURNING id;  -- capture the new id, call it $new_id

UPDATE semantic_facts
SET superseded_by = $new_id,
    superseded_at = now()
WHERE subject = $1
  AND predicate = $2
  AND superseded_at IS NULL
  AND id <> $new_id;

COMMIT;

The subject+predicate match is the supersession key — "user 42's preferred deploy target" is a single slot in the conceptual model, and only one row should be live per slot at a time. The transaction is what makes this safe under concurrent agent ticks: the partial index sees a consistent live-view at any point.

Retention and bounded growth

Without retention every memory table grows unboundedly. Three rules covers the realistic shape:

Episodic older than 30 days, except events referenced by a still-live semantic_facts.source_event. A scheduled DELETE with a NOT EXISTS subquery does the work.
Semantic with superseded_at < now() - interval '180 days' is hard-deleted. Soft-supersedes preserve recent history; long-tail history goes.
Procedural with expires_at < now() is deleted at a fixed cadence, or on eviction pressure (track pg_relation_size against a budget).

This is the difference between a memory layer and a memory leak. Without these, the agent's recall gets slower every week and you eventually rewrite it as a sibling vector store with a CDC pipeline — which is exactly the architecture the multi-model approach exists to avoid.

What this schema is not

This is not a replacement for a working-set context buffer (the last few thousand tokens of the current conversation, held in memory or in the agent process's own state). That belongs in the agent runtime, not the database. Episodic memory is the durable record of the same conversation, available across sessions and across agents.

It also isn't a replacement for vector-only embedding databases at very high cardinality — if you have a billion fact rows and need single-digit-millisecond ANN, the access pattern looks different. The schema here is sized for the typical agent project: thousands to millions of events, tens of thousands to low millions of facts. That is where most projects live for the first two years, and it is exactly the size where the operational overhead of a second database hurts the most.

The shorter version

Three tables, separated by access pattern not by data type:

episodic_events — ordered, mostly immutable, nullable embedding, retained for weeks.
semantic_facts — soft-superseded, subject-keyed, always embedded, retained for months.
procedural_cache — hash-keyed, explicit TTL, hit-counted, retained until expiry.

One transaction writes an event plus any extracted facts. One query reads the conversation; one query reads the subject's live facts; one query reads the cache. The drift window is gone because the writes are atomic. The supersession history is preserved because updates aren't destructive. Retention is bounded because every table has an answer to "when do rows leave."

Copy the schema. Tune the embedding dimensions, the retention intervals, the HNSW parameters. The shape stays.

Self-hosting RedDB: the real-world checklist

2026-04-25

This is the post we wish existed when we first put RedDB on a customer's cluster. It is not a marketing surface — it is the literal checklist our SRE walks through before declaring an environment "production-ready." If you can answer every item below with "yes, and here's the evidence," you have a self-host that will not page you at 03:00.

Everything here assumes Kubernetes because that is where ~90% of self-hosters land. The same shape applies on plain VMs with systemd or on Nomad; the artifacts change, the decisions do not.

The five-minute version

Run this checklist before you put traffic on the cluster:

StatefulSet, not Deployment, for the data plane.
PersistentVolumeClaim sized at 3× expected hot working set, on SSD-class storage with fsync honored.
readinessProbe that checks the WAL is fsynced and a follower has caught up to within 5 seconds.
Prometheus scrape against /metrics on the admin port (not the data port).
A CronJob that runs a backup against object storage, with retention >= compliance window + 7 days.
NetworkPolicy that only allows the application namespace, the monitoring namespace, and the backup job to reach the data port.
KMS-backed secret for the encryption-at-rest key (see vault key rotation for why this matters).
A documented restore drill that someone ran in the last 90 days.

The rest of this post is the long form of those eight bullets.

Container image: pin, don't `:latest`

image: registry.reddb.io/reddb:1.8.4
imagePullPolicy: IfNotPresent

Two reasons we belabor this. First, :latest defeats the whole point of immutable infra — a node reschedule silently rolls you forward. Second, our minor versions occasionally change defaults (compaction priority, bloom sizing) and you want the choice to be deliberate. Pin the patch, read the changelog before you bump the minor.

Image size is ~80 MiB compressed. We ship a -debug variant with pprof and a shell; do not use it as your default runtime — it has a strictly larger attack surface for no operational benefit.

StatefulSet, not Deployment

The data plane is stateful. The right Kubernetes primitive is StatefulSet. A working manifest looks like:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: reddb
  namespace: reddb-system
spec:
  serviceName: reddb-headless
  replicas: 3
  podManagementPolicy: Parallel
  selector:
    matchLabels:
      app: reddb
  template:
    metadata:
      labels:
        app: reddb
    spec:
      terminationGracePeriodSeconds: 120
      securityContext:
        runAsUser: 10001
        runAsGroup: 10001
        fsGroup: 10001
      containers:
        - name: reddb
          image: registry.reddb.io/reddb:1.8.4
          args: ["--config=/etc/reddb/config.toml"]
          ports:
            - name: data
              containerPort: 5432
            - name: replica
              containerPort: 5433
            - name: admin
              containerPort: 9180
          env:
            - name: REDDB_NODE_ID
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: REDDB_KMS_KEY_ARN
              valueFrom:
                secretKeyRef:
                  name: reddb-kms
                  key: arn
          volumeMounts:
            - name: data
              mountPath: /var/lib/reddb
            - name: config
              mountPath: /etc/reddb
              readOnly: true
          resources:
            requests:
              cpu: "2"
              memory: 8Gi
            limits:
              memory: 12Gi
          readinessProbe:
            httpGet:
              path: /readyz
              port: admin
            periodSeconds: 5
            failureThreshold: 3
          livenessProbe:
            httpGet:
              path: /livez
              port: admin
            periodSeconds: 30
            failureThreshold: 3
            initialDelaySeconds: 60
      volumes:
        - name: config
          configMap:
            name: reddb-config
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: gp3-fsync-honored
        resources:
          requests:
            storage: 200Gi

A few decisions inside this we want to call out, because the defaults regularly trip people:

podManagementPolicy: Parallel. The default is OrderedReady, which serializes startup. RedDB nodes form a quorum on boot — they need to come up roughly together, not one after the other waiting for the previous to be Ready.
terminationGracePeriodSeconds: 120. The data plane needs time to flush the WAL, hand off the leader role if it owned one, and close client connections cleanly. 30 seconds (the default) will cause WAL replay on restart, which adds startup latency and shows up as a recurring availability dip.
CPU request 2, no limit. Compaction is bursty. Setting a CPU limit will cause throttling during compaction windows and you will see the exact latency spike that the LSM compaction notes describe. Memory limits, however, are fine and recommended — RedDB respects the cgroup memory limit and shrinks its block cache accordingly.
livenessProbe.initialDelaySeconds: 60. WAL replay on a cold node can take a minute. Without the delay, a slow-starting node gets killed mid-replay, replays again, gets killed again. We have seen production clusters stuck in this loop for hours.

Storage: `gp3-fsync-honored`

This is the single line item that decides whether your self-host is durable or theatrical. RedDB's durability guarantees assume fsync actually persists data before returning. Many cloud storage classes lie about this for performance. The classes that do not lie, by cloud:

AWS: gp3 (set iopsPerGB to at least 3, and confirm the EBS-CSI driver is not running with dataPlaneOptions.fsyncDisabled).
GCP: pd-ssd (the default pd-standard does honor fsync but the IOPS profile is unsuitable).
Azure: Premium SSD v2 (the older Premium_LRS works but costs more for less).
Bare metal: NVMe with LITTLE_ENDIAN write barriers enabled in the filesystem (ext4 with data=ordered, the default, is correct; data=writeback is not).

Size the PVC at 3× expected hot working set. The 3× accounts for: 1× the live data, ~0.5× the SSTables waiting on compaction, ~0.5× temporary space for compaction itself, and the rest as headroom for pgbench-style load tests and the occasional bloat. Resizing PVCs is supported but disruptive; oversize on day one.

Monitoring: the four metrics that matter

RedDB exports ~120 Prometheus metrics. Watch these four; alert on the first three:

Metric	What it tells you	Page threshold
`reddb_wal_fsync_seconds{quantile="0.99"}`	Storage subsystem health	> 50ms
`reddb_replication_lag_bytes{follower=~".*"}`	Follower divergence	> 100 MiB sustained 5min
`reddb_compaction_score{level=~".*"}`	Compaction debt	> 50 sustained 10min
`reddb_block_cache_hit_ratio`	Working set fits	(track, don't page)

If wal_fsync_seconds spikes, your storage is the problem, full stop. Do not start tuning RedDB until that number is back under 10ms. We have spent days helping customers tune things that were not the bottleneck because they were watching the wrong dashboard.

If compaction_score is climbing without bound, you are writing faster than the cluster can compact. The fix is either more storage IOPS, more CPU, or write backpressure on the application side. RedDB has a built-in backpressure mechanism that returns SQLSTATE 53300 (cannot connect now); turn it on with compaction.backpressure.enabled = true.

Backups: object storage, encrypted, drilled

The backup job we recommend is a CronJob that triggers RedDB's built-in pg_basebackup-equivalent:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: reddb-backup
  namespace: reddb-system
spec:
  schedule: "0 3 * * *"
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 7
  failedJobsHistoryLimit: 14
  jobTemplate:
    spec:
      backoffLimit: 2
      template:
        spec:
          restartPolicy: OnFailure
          serviceAccountName: reddb-backup
          containers:
            - name: backup
              image: registry.reddb.io/reddb-backup:1.8.4
              args:
                - --source=reddb-0.reddb-headless.reddb-system.svc:5432
                - --destination=s3://acme-reddb-backups/$(NODE)/$(DATE)/
                - --encrypt-key-arn=$(BACKUP_KMS_ARN)
                - --retention-days=35
              env:
                - name: NODE
                  value: reddb-0
                - name: DATE
                  valueFrom:
                    fieldRef:
                      fieldPath: metadata.name
                - name: BACKUP_KMS_ARN
                  valueFrom:
                    secretKeyRef:
                      name: reddb-backup-kms
                      key: arn

Three things to verify in your environment:

Backups land where you think. Run aws s3 ls s3://acme-reddb-backups/ --recursive | head the morning after the first run. A backup job that exits 0 but writes to the wrong prefix is the textbook silent failure.
Retention matches compliance. --retention-days=35 covers a 30-day compliance window with a week of buffer. If you are in a 7-year regulated industry, you want a tiered policy (daily for 30 days, monthly for 7 years) — the simplest implementation is two separate CronJobs writing to two separate prefixes with two separate lifecycle rules on the bucket.
You have done a restore drill recently. Backups that have never been restored are wishes. Once per quarter, restore to a scratch cluster, run a smoke query, write down the wall-clock time. If your RTO target is 1 hour and your last drill took 2 hours, you have a problem you would rather discover now than during the incident.

Network policy: deny-by-default, then carve out

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: reddb-data-plane
  namespace: reddb-system
spec:
  podSelector:
    matchLabels:
      app: reddb
  policyTypes: [Ingress]
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              app.kubernetes.io/name: application
      ports:
        - protocol: TCP
          port: 5432
    - from:
        - namespaceSelector:
            matchLabels:
              app.kubernetes.io/name: monitoring
      ports:
        - protocol: TCP
          port: 9180
    - from:
        - podSelector:
            matchLabels:
              job-name: reddb-backup
      ports:
        - protocol: TCP
          port: 5432
    - from:
        - podSelector:
            matchLabels:
              app: reddb
      ports:
        - protocol: TCP
          port: 5433

The deny-by-default is the important bit. Almost every incident we have helped with where someone exfiltrated data started with "the data port was reachable from somewhere we did not realize." Carve out the three flows the cluster actually needs (app → data, monitoring → admin, backup-job → data, replica → replica) and nothing else.

Secrets: KMS-backed, not literal Kubernetes Secrets

The encryption-at-rest key is the one secret that, if leaked, ruins your night even if you rotate it immediately (because the leaked version covers everything written under it). Treat it accordingly:

Store the KMS ARN in a Kubernetes Secret, not the key material itself. RedDB unwraps at boot via the pod's IAM role / workload identity.
Rotate the KMS key on the documented schedule (we walked through the zero-downtime mechanism in the vault rotation post).
Audit access to the KMS key, not just access to the cluster. Most Kubernetes audit dashboards do not surface "someone called kms:Decrypt against the production key from outside the cluster's IAM role," which is exactly the alert you want.

Things this checklist deliberately skips

A few topics that come up and that you should think about separately, not as part of "is the cluster ready":

Multi-region. The above is for a single region. Multi-region is the topic of the disaster-recovery post — RPO/RTO trade-offs, async vs synchronous replication, and the operational tax both impose.
Connection pooling. Run a pooler (PgBouncer, RedDB's bundled reddb-pool) between your app and the data plane. Sizing the pool is a function of your app, not the cluster.
Schema migrations. Use whatever your team already uses (Atlas, sqitch, plain psql scripts in CI). The cluster does not care, and any opinion this post had would just be cargo-culted on day one.
Capacity planning. Out of scope for "get to production-ready." Re-evaluate every quarter after you have real traffic.

TL;DR

Pin the image, use a StatefulSet, give it fsync-honoring SSD storage at 3× hot working set, watch four metrics, run encrypted backups to object storage and actually restore them, deny-by-default networking, KMS-back the at-rest key. If you can show evidence for each of those, you are production-ready. If you cannot, the cluster will tell you which one you skipped — usually around 03:00.

The companion DR post picks up from here: what to do when the region itself goes away.

Why 'best of breed' loses at small scale

2026-04-24

The "best of breed per workload" line has a comforting shape: pick the perfect tool for each job, glue them together, ship faster than the team that picked one general-purpose engine. It reads as engineering maturity. In practice, at the team sizes where most products live — between two and fifteen engineers — it is a tax that almost nobody finishes paying.

This post tries to put a number on that tax. Not "it feels slower" but person-hours per quarter, per engine, per team size. The numbers below are not from a benchmark; they are the rolling average of what we and our design partners have observed across teams running between two and seven storage engines in production.

The stack we are measuring

The "best of breed" reference stack:

Postgres for relational rows
Elasticsearch (or OpenSearch) for full-text search
A vector store (Pinecone, Weaviate, or pgvector-on-a-second-cluster) for RAG
Redis for sessions, rate limits, and a queue

Four engines. Each is, in isolation, a strong choice. The question this post is asking: what does the seam between them cost, and how does that cost scale with team size?

The single-engine alternative: one multi-model store (RedDB, or for the sake of argument any system that handles the four shapes under one transaction and one operator surface).

The hidden line items

Before the table, the hidden costs people forget when they sketch the "best of breed" decision:

Training. Each engine has its own query language, failure modes, tuning knobs, backup procedure. A new hire is not productive on the stack until they can debug all of them. With one engine, "productive on the database" is one onboarding milestone; with four, it is four.
On-call rotation per system. If you take on-call seriously, every engine in production has runbooks, alerts, and a person who knows it well enough to wake up at 3am. With four engines, either four people each own one (and the bus factor is one across the stack) or every on-call person owns four (and the runbook reading list is long).
IAM sprawl. Every engine has its own auth model, its own service accounts, its own audit log to wire into the SIEM. Four times the IAM surface is four times the place a misconfigured policy hides.
Cross-engine consistency code. This is the line item nobody budgets for. When the source-of-truth row in Postgres updates and the search index in Elastic and the embedding in the vector store have to follow, somebody writes — and maintains, forever — the outbox, the queue, the retry policy, the dead-letter handler, the reconciliation job. Per pair of engines. With four engines and three data flows between them, that is three independent pieces of glue, each with its own failure mode.
Backup correlation. A backup of Postgres taken at 02:00, an Elastic snapshot taken at 02:15, and a Pinecone export from yesterday are not a coherent point-in-time view of your system. Restoring is a science project. With one engine, the backup is the backup.
Local dev parity. Each engine needs to run on the developer's laptop or be mocked. With four, the docker-compose file becomes a load-bearing artifact and "it works locally" stops being a useful signal.

None of these costs show up in the per-engine pricing page. All of them show up in the team's calendar.

The ops cost table

Person-hours per quarter, by engine, by team size. "Engine" here means one production storage system the team owns end-to-end. "Glue" is the cross-engine consistency code and reconciliation work.

Activity	Per engine, 2-person team	Per engine, 5-person team	Per engine, 15-person team
Patching + version upgrades	8 h	12 h	20 h
Backup verification + restore drill	6 h	10 h	16 h
Capacity planning + cost review	4 h	8 h	14 h
Runbook maintenance + on-call training	4 h	10 h	22 h
Incident response (avg, amortized)	12 h	18 h	28 h
Schema/index/config change reviews	6 h	14 h	30 h
Per-engine subtotal	40 h	72 h	130 h

And the cross-engine line items:

Activity	2-person	5-person	15-person
Glue maintenance (per pair of engines)	8 h	14 h	22 h
Cross-engine incidents	10 h	16 h	30 h
Schema drift reconciliation	6 h	12 h	20 h

Running the math

Best-of-breed stack: four engines, three glue pairs.

Team size	Per-engine cost (×4)	Glue cost	Total q/quarter	% of team capacity
2	160 h	72 h	232 h	~48 % of one engineer
5	288 h	126 h	414 h	~34 % of one engineer
15	520 h	216 h	736 h	~20 % of one engineer

Single-engine stack: one engine, no glue.

Team size	Per-engine cost	Total q/quarter	% of team capacity
2	40 h	40 h	~8 % of one engineer
5	72 h	72 h	~6 % of one engineer
15	130 h	130 h	~3.5 % of one engineer

Assumes one engineer-quarter = ~480 productive hours (60 working days × 8 h, lightly discounted for meetings and PRs).

What the table says, in English

At two engineers, the best-of-breed tax is roughly half of one engineer, every quarter, forever. That is a co-founder. Half of one of two people is doing nothing but keeping the seams between engines from tearing. If those two people were trying to find product-market fit, they have just lost a quarter of the company's total capacity to glue.

At five engineers, it is closer to a third of a person — but a five-person team that gives up a third of a person to undifferentiated ops is one that is not shipping the thing customers are asking for. The team feels it as "we are always firefighting." That is what the table is measuring.

At fifteen engineers, the percentage drops to a fifth of a person — but the absolute number is 736 hours per quarter, the equivalent of more than four engineer-months annually. At fifteen people, that often pays for the dedicated platform team that the team needed in order to run the stack. The cost has not gone away; it has been institutionalized.

The single-engine column, by contrast, scales sub-linearly. There is no glue, so adding engines to the stack does not add the worst line item. The increase from a two-person to a fifteen-person team is roughly 3× — and most of that is "more engineers means more change reviews," not more ops surface.

"But we're not paying that cost"

Three common objections, with what we have seen in practice.

"Our cloud-managed services handle ops." Partly. Managed Postgres, managed Elastic, managed Redis, managed vector store — each one removes the patching line item and some of the capacity planning. It does not remove the runbooks, the IAM sprawl, the cross-engine glue, the backup correlation, the on-call surface, or the dev-parity problem. Maybe halve the per-engine row. The glue row does not budge, because that is your code, not the provider's.

"We have one person who owns each engine." This is the bus-factor-one reframing of the cost, not its elimination. The work is the same; it has been concentrated on individuals whose departure will be load-bearing. It also concentrates expertise where it cannot be reviewed, which is its own incident waiting to happen.

"We do not have all four engines yet." Then the calculation is the one you should be doing now, before adding the third. The pattern we have seen is that engines are added one at a time, each addition is locally justified ("we just need full-text search"), and the team realizes only at engine four that the integration tax has compounded. Each additional engine adds another glue pair, not a constant.

Where the framing breaks

This is not a universal claim. There are real reasons to run a best-of-breed stack:

The single-engine option is genuinely worse at one of your shapes. If you need true OLAP — billion-row scans, columnar compression — a multi-model engine is not the right tool. ClickHouse exists for a reason.
You have a regulatory requirement that names a specific engine. Use the named engine; the auditor's report is not the place to be clever.
One of the shapes is so high-volume that it dominates the cost. If 95 % of your read traffic is Redis-shaped (millions of cache hits per second), running Redis separately and keeping it tuned is genuinely cheaper than putting that load on a general-purpose engine.

These are real cases. They are also the cases where, at any team size, the math works out. Most teams do not have them. Most teams have four engines because each was added under local pressure, not because the stack was designed.

Multi-model as a small-team unlock

The framing this piece is most arguing against: "multi-model is for scale." The table says the opposite. The team that benefits most from one engine instead of four is the two-person team, because the absolute ops cost is the same fraction of a much smaller capacity. A 200-engineer company can run seven engines and not feel it. Two co-founders cannot.

This reframes when to consider multi-model. Not when you grow into it; before you grow. The choice to run one engine instead of four is one of the highest-leverage decisions a small team makes, and almost nobody frames it that way at the moment of decision, because each engine looks free in isolation.

We are not the only multi-model option. Postgres, with its growing extension ecosystem, covers more shapes every year and is a reasonable single-engine answer for many teams — we said as much in When you don't need RedDB. The point of this post is not "pick RedDB." It is "pick one engine, whichever one matches your shapes."

TL;DR

Person-hours per quarter on the ops/glue surface of a four-engine stack: ~232 h at 2 people, ~414 h at 5, ~736 h at 15.
Same calculation for a one-engine stack: ~40 h, ~72 h, ~130 h.
The percentage cost is highest at the smallest team — multi-model is a small-team unlock, not a scale concern.
Managed services reduce the per-engine row, not the glue row. The glue is your code.
Where best-of-breed wins: true OLAP, named regulatory requirements, one workload dominates the cost. Most teams do not have these.

When you don't need RedDB

2026-04-22

We sell a database. Telling you when not to buy it is, on the face of it, a strange use of a marketing surface. We do it anyway because the alternative — watching someone migrate onto RedDB, hit a wall that was visible from a mile away, and churn six months later — is a worse outcome for everyone. A reader who saves a migration they would have regretted is a reader who trusts the next post.

This piece is the list. If your shape is here, the right answer is "stay where you are" or "pick the boring single-purpose store." We will be specific about which one.

The frame: what RedDB is actually for

RedDB earns its keep when a single workload needs more than one storage shape at the same time. Relational rows with vector search next to them. Documents with full-text search next to them. Event log with derived materialized views next to it. The pitch is that the second shape is in the same transaction, the same auth model, the same backup, the same operator surface. If you do not have two shapes — or you have two but they never need to be consistent with each other — most of that value evaporates and you are paying the cost of a less-mature engine for no return.

Everything below is a variant of "you only have one shape, or your two shapes never need to be consistent."

1. Single-modality OLTP with no vectors and no documents

If your data model is "ten well-normalized tables, some foreign keys, a handful of indexes, and the hardest query is a three-way join," use Postgres. Not pgvector, not a multi-model anything — just Postgres.

Postgres at this shape is the boring right answer for reasons that have very little to do with the database itself:

Every cloud has a managed Postgres with a 10-year operational track record. The on-call story is solved without you doing anything.
Every ORM, every BI tool, every analyst-facing dashboard speaks Postgres natively. You do not have to argue with anyone about driver support.
Every senior engineer you will ever hire has debugged a Postgres query plan before. Ramp-up is zero.

The RedDB pitch in this shape reduces to "maybe slightly different operational defaults." That is not enough to justify being the one team in your company on a less-deployed engine. Stay on Postgres.

2. Embedded / single-user / single-process workloads

If your workload runs in a desktop app, a mobile app, a CLI, an edge device, or a serverless function with a local writable disk, use SQLite. (For mobile-first sync, Turso or Cloudflare D1 sit on top of the same engine and add the sync layer.)

SQLite is not "Postgres lite." It is a fundamentally different operating model — in-process, zero servers, the file is the database. RedDB is a server process with a network protocol; running it embedded would be a category error. The minute your data fits this shape, every "but RedDB has feature X" is irrelevant because the cost of running a server next to your app dwarfs the value of feature X.

Heuristic: if your "production deployment" is "ship a binary that touches a local file," SQLite is the answer.

3. Workloads where the second shape never needs to be transactionally consistent

This is the subtle one. Just having "two shapes" does not mean you need a multi-model database. The honest question is: do the two shapes need to be consistent with each other on every write?

Take the classic "I want logs and metrics next to my OLTP data." You almost certainly do not need those to be in the same transaction as your business writes. The log write losing a record because of a transient downstream outage is fine. The metric being a few seconds stale on a dashboard is fine. Stream them into the dedicated tool (Loki, Prometheus, ClickHouse) and let the OLTP database be the OLTP database. The cost of putting them in the same engine is real; the consistency benefit is zero because no consumer of those streams expects strict consistency.

A useful test: if you can imagine an outage where one shape lags the other by 30 seconds and no user ever notices, you do not need transactional consistency between them, and you do not need a multi-model database. You need two databases and an event bus.

The RedDB shape that earns its keep is the inverse — the search index that lags the document by 30 seconds will show a user a result that no longer matches the source. Drift visible to users is the bar.

4. Hyper-low write volume with no operational team

If you are a two-person team running a marketing site, an internal tool, or a side project that does fewer than ~1 write per second on a peak day, the operational overhead of running any dedicated database is larger than the storage problem you are solving.

Concretely: use the database your hosting provider offers in the same control plane as your app. Supabase Postgres, Vercel Postgres, Neon, PlanetScale, Cloud SQL — whichever is one click from where your app already runs. You will not benefit from RedDB's multi-model story at this volume because you do not have time to build a system that uses the multi-model story. You have time to ship features, and a managed Postgres with zero ops on your plate is what enables that.

A subtler version of this rule: even if you do have two shapes (you want vector search for an AI feature), if you are a two-person team, use Postgres + pgvector until you outgrow it. It is one engine, one backup, one connection pool. The drift-window problem is real but tolerable at low write volume, and you will know when you have outgrown it because users will start complaining about stale search results.

5. Teams without ops appetite for a less-mature engine

This is the most uncomfortable item on the list, so we will be direct: RedDB is younger than Postgres. It has fewer years of production deployments, fewer post-mortems published, fewer Stack Overflow answers for obscure error codes. The engineering is solid, the test coverage is good, the public benchmarks hold up — but maturity is a function of calendar time and number of operators, and there is no shortcut.

If your organization treats database choice as a near-irreversible commitment, if your security team requires three years of public CVE history before approving an engine, if your on-call rotation does not have one engineer who is comfortable reading source code when a tool misbehaves — those are entirely reasonable constraints, and the rational choice under them is the most-deployed engine in your category. Pick Postgres. Pick the cloud-native option from your provider. Revisit RedDB in 18 months when more deployments have shaken out the edges.

We would rather you make that call clear-eyed than have you adopt RedDB, hit an operational surprise, and conclude the engine is unfit when the actual mismatch was risk appetite.

6. Workloads that are really about analytics, not transactions

If the dominant query pattern is "scan large slices of historical data and aggregate," you have an analytical workload, and the right answer is an analytical engine: ClickHouse, DuckDB, BigQuery, Snowflake, depending on your scale and budget. Putting analytical scans on a transactional engine — any transactional engine, RedDB included — gets you a much worse cost-per-query than the dedicated tool, and you lose the columnar storage and vectorized execution that make analytical engines fast.

The mixed shape (some OLTP, some analytical) is real and is one of the places RedDB is genuinely interesting, but the threshold is "the analytical queries are a minority that need to see fresh transactional data." If your analytical queries are the majority of your workload and they can tolerate a 5-minute replication lag from a source-of-truth OLTP store, the two-engine answer (Postgres → ClickHouse via CDC) is cheaper and more performant.

7. You already have the four engines and they are working

If you are currently running Postgres + Elasticsearch + Pinecone + Redis and the team is happy, the on-call is quiet, and your latency numbers are within budget — do not migrate. The "small-team unlock" story for RedDB (we wrote a separate post on the ops math) is about teams who would otherwise adopt four engines. The cost of migrating an already-working four-engine stack is almost never recovered by the operational simplification, because the migration itself burns the budget that would have funded the simplification.

The rule of thumb we use internally: a migration earns its budget back if it is paying down a problem the team is currently losing sleep over. If nobody is currently losing sleep, leave the stack alone and spend the engineering time on features.

8. Strict regulatory requirements naming a specific engine

A small number of regulated industries have certifications that name specific database engines, or specific cloud-managed offerings, in the compliance scope. If your auditor's checklist says "Postgres with FedRAMP-authorized hosting" and RedDB is not on that list, the cost of getting it added to the authorized boundary exceeds the value, period. Use the named engine.

This is not a knock on the engine — it is a recognition that compliance scopes are slow-moving by design, and a database choice is not the right place to spend novelty budget when an auditor is in the loop.

The shapes where RedDB does earn its keep

For balance, here is the inverse list — shapes where the multi-model story actually pays off:

RAG workloads where the embedding must stay consistent with the source document on every write (the drift-window problem). Long-form version of the argument here.
Agent memory layers where episodic events, semantic facts, and procedural caches are written and read together in the same request.
Search-over-mutable-data products where the search index lagging the source by even seconds is a user-visible bug.
Multi-tenant SaaS with per-tenant isolation where running N copies of four engines is cost-prohibitive but per-tenant rows in one engine is tractable.
Small teams (2–5 engineers) who would otherwise stand up four engines — the ops cost math flips here, not at high scale.

If you are not in one of those shapes, the honest answer is the boring one. Save the migration for a problem that needs it.

TL;DR

Stay on what you have if:

One storage shape, no vectors, no documents → Postgres.
Embedded / single-process → SQLite.
Two shapes that never need to be consistent → two databases and an event bus.
Two-person team, tiny write volume → whatever your hosting provider runs.
Risk-averse org with hard maturity requirements → Postgres.
Analytics-dominant workload → ClickHouse / DuckDB / BigQuery.
Four engines that are working and quiet → leave them alone.
Auditor named a specific engine → use the named engine.

We will be here when one of the inverse shapes shows up.

RedDB blog

RedDB on a Raspberry Pi: the smallest sane install

What you need

Image and install

Load a dataset

Run a query

What this is and is not

Cross-session task queues — AFK agents with durable resume

The AFK promise that breaks at 3am

The schema: state, checkpoint, idem_key

Enqueue: idempotent and ordered

Claim: SKIP LOCKED + lease

Checkpoint: write often, write cheap

Resume: read the checkpoint, don't redo it

Watchdog: reclaim, retry, give up

Stop handler: graceful pause, not a crash

When to skip this

Tying it together

Eval datasets in RedDB — from ad-hoc prompts to versioned regression sets

The notebook eval is a dead end

The schema: cases, datasets, runs

Capture: the hook that fills the table

Publish: freezing a dataset

Replay: the eval is a query

The eval is the SQL afterwards

When this is overkill

Sub-agent dispatch via database queue — durable parallelism across sessions

The Task tool's shelf life is one session

The schema: one table, three states, a notify trigger

Dispatcher: a tiny wrapper around Task

Worker: claim, run, write back

Watchdog: reclaim leases, alert on poison pills

Calling it from a Claude Code session

When this pattern earns its keep

What's next

Context window economics — when memory beats reloading docs

The hidden bill on every CLI agent session

The naive baseline: re-read everything

The memory-backed alternative

The breakeven, plotted as algebra

The actual code

What to measure once it's live

When this is the wrong call

Where this fits in the pillar

Multi-agent shared memory — five agents, one knowledge graph

Why parallel agents need a shared scratchpad

The shape of the fix

Schema

The write path

The read path

Recording conflicts and alerting the orchestrator

Promoting consensus

Worked example

What this gives you

Agent observability — traces, tokens, and costs in RedDB

The questions you can't answer today

The schema

The PostToolUse hook

The rollup trigger

The five queries that earn the hook

A dashboard sketch

What this buys you

What is next in this pillar

An MCP server for RedDB — one database, every agent CLI

Why MCP is the right seam

The manifest

The server

Wiring into Claude Code

What you gave up, and what you bought

Next

Building a "remember this" skill end-to-end

What we're building

Prerequisites

File 1 — SKILL.md

Do not

File 3 — nothing (read-back already exists)

Testing it

What goes wrong in week one

Why this shape

Next

File 1 — `SKILL.md`

`/remember` — the write

`/recall` — the read

`/forget` — the prune

1. Record installations on `SessionStart`

2. Log every run via `PostToolUse`

3. Build skill edges in a `Stop` hook

A `/suggest-skill` slash command

Mitigation 3 — `wal_priority` hint on the write path