Agents · 2026-06-05 · By RedDB team · 4 min read

Eval datasets in RedDB — from ad-hoc prompts to versioned regression sets

Most teams evaluate agents with a notebook full of one-off prompts that nobody re-runs. Capture every interesting turn into a RedDB table, tag it, version it, and your eval becomes a query that any model upgrade can be diffed against in seconds.

The notebook eval is a dead end

Pick any team running an agent in production for more than a quarter and look at how they evaluate it. You will find a Jupyter notebook, an evals.py, or a folder of .txt prompts. Someone wrote them the week the agent shipped. Nobody has re-run them in two months. When a new model drops — Claude Opus 4.7 last month, Sonnet 4.6 the week before — the same notebook gets opened, a few cells get edited, results are eyeballed, “looks fine,” done.

That is not an eval. That is a memory of an eval.

The problems compound:

  • No baseline. The output of last month’s run is gone. You cannot diff.
  • No coverage signal. Which production scenarios are covered? No one knows. The prompts that matter — the ones that broke last quarter — are not in the file.
  • No tagging. “Show me the long-context cases” requires grep over a folder.
  • No fresh data. Every interesting failure the agent had this sprint is in a Slack thread, not in the eval set.

The fix is the one a database engineer would write on a napkin: every interesting interaction is a row, tagging is a column, versioning is a dataset_version table, replay is a SELECT … WHERE tag = 'long-context', and the model upgrade is a JOIN between two run tables. Builds on the memory schema from D1, the observability tables from D9, and the dispatch queue from D12 (workers replay the set in parallel).

The schema: cases, datasets, runs

Three tables. Cases are the facts. Datasets are versioned snapshots. Runs are the result of replaying a dataset against one model + prompt revision.

-- A single captured interaction. Append-only.
CREATE TABLE eval_case (
  id              uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  source          text NOT NULL,                  -- 'production' | 'incident' | 'handcrafted'
  prompt          text NOT NULL,                  -- full user turn, including system context
  context         jsonb NOT NULL DEFAULT '{}',    -- files, tool history, env
  reference       text,                           -- gold answer if known; NULL = judged at run time
  tags            text[] NOT NULL DEFAULT '{}',   -- ['long-context','tool-use','regression-#1284']
  captured_at     timestamptz NOT NULL DEFAULT now(),
  captured_by     text NOT NULL,                  -- agent id or user
  embedding       vector(1024)                    -- voyage-3, for "find similar cases"
);

CREATE INDEX eval_case_tags_idx  ON eval_case USING gin (tags);
CREATE INDEX eval_case_embed_idx ON eval_case USING hnsw (embedding vector_cosine_ops);

-- A frozen, named snapshot of cases. Immutable once published.
CREATE TABLE eval_dataset (
  id          uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  name        text NOT NULL,                      -- 'core-v3', 'long-context-2026-q2'
  version     int  NOT NULL,
  case_ids    uuid[] NOT NULL,                    -- locked at publish
  notes       text,
  published_at timestamptz NOT NULL DEFAULT now(),
  UNIQUE (name, version)
);

-- One row per (dataset_version, case, model+prompt revision) replay.
CREATE TABLE eval_run (
  id            uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  dataset_id    uuid NOT NULL REFERENCES eval_dataset(id),
  case_id       uuid NOT NULL REFERENCES eval_case(id),
  model         text NOT NULL,                    -- 'claude-opus-4-7'
  prompt_rev    text NOT NULL,                    -- git sha of the system prompt
  output        text NOT NULL,
  pass          boolean,                          -- NULL if judged later
  judge_score   numeric(3,2),                     -- 0.00–1.00
  judge_reason  text,
  latency_ms    int  NOT NULL,
  input_tokens  int  NOT NULL,
  output_tokens int  NOT NULL,
  cost_usd      numeric(10,6) NOT NULL,
  ran_at        timestamptz NOT NULL DEFAULT now()
);

CREATE INDEX eval_run_dataset_idx ON eval_run (dataset_id, model, prompt_rev);

Three design choices to call out:

  1. Cases are append-only. A captured case never mutates. If the gold answer was wrong, you write a new case and tag the old one superseded. This is how you get clean diffs across versions.
  2. Datasets freeze case_ids. Publishing a dataset is a snapshot. Six months from now you can rerun core-v3 and the membership has not drifted.
  3. prompt_rev is the git SHA of the system prompt. Without it, a regression looks like a model regression when it is actually a prompt edit. Always log both.

Capture: the hook that fills the table

Capturing has to be cheap or nobody does it. A PostToolUse hook with a slash-command escape hatch covers both flows:

  • Auto-capture any turn that ends with an explicit user thumbs-down (/eval-bad) or a hook-detected failure (timeout, retry-exhausted, judge score < threshold).
  • Manual capture via /eval-capture <tag1,tag2> when the human notices something worth keeping.

/eval-capture is one shell script. It reads the current conversation tail, embeds the prompt with voyage-3, and INSERTs:

#!/usr/bin/env bash
# ~/.claude/commands/eval-capture.sh
set -euo pipefail

tags="${1:-handcrafted}"
prompt="$(claude-cli conversation tail --turns 1 --format text)"
context="$(claude-cli context dump --format json)"

embedding="$(curl -s https://api.voyageai.com/v1/embeddings \
  -H "Authorization: Bearer $VOYAGE_API_KEY" \
  -H 'Content-Type: application/json' \
  -d "$(jq -nc --arg i "$prompt" \
        '{model:"voyage-3", input:[$i]}')" \
  | jq -c '.data[0].embedding')"

psql "$REDDB_URL" -v ON_ERROR_STOP=1 <<SQL
INSERT INTO eval_case (source, prompt, context, tags, captured_by, embedding)
VALUES (
  'handcrafted',
  \$\$${prompt//\$\$/\$\$\$\$}\$\$,
  '${context}'::jsonb,
  string_to_array('${tags}', ',')::text[],
  '${USER}',
  '${embedding}'::vector
);
SQL

echo "captured to eval_case (tags: ${tags})"

PostToolUse is the same idea wrapped around a failure detector — drop in the transactional-hook pattern from D3 so a half-written embedding never lands without its row.

Publish: freezing a dataset

A dataset is a one-line query plus an insert:

-- Pin everything captured this quarter that is not superseded.
WITH picked AS (
  SELECT id FROM eval_case
  WHERE captured_at >= '2026-04-01'
    AND NOT ('superseded' = ANY(tags))
)
INSERT INTO eval_dataset (name, version, case_ids, notes)
SELECT 'core', 4,
       array_agg(id ORDER BY captured_at),
       'Q2 2026 baseline, includes the 1284 long-context regression'
FROM picked;

Membership is now frozen. Anyone — six months, three model upgrades, two prompt rewrites from now — can rerun core v4 and get the same case set.

Replay: the eval is a query

Replay is a function over (dataset_id, model, prompt_rev). The naive single-process version is fine for ~200 cases; for anything bigger, push each case into the dispatch queue from D12 and let N workers chew through it in parallel.

// scripts/replay.ts
import { Client } from 'pg'
import Anthropic from '@anthropic-ai/sdk'

const db  = new Client({ connectionString: process.env.REDDB_URL })
const ai  = new Anthropic()
const MODEL       = process.env.MODEL       ?? 'claude-opus-4-7'
const PROMPT_REV  = process.env.PROMPT_REV  ?? execSyncSha('prompts/system.md')
const DATASET     = process.env.DATASET     ?? 'core'
const VERSION     = Number(process.env.VERSION ?? 4)

await db.connect()

const { rows: [ds] } = await db.query(
  `SELECT id, case_ids FROM eval_dataset WHERE name=$1 AND version=$2`,
  [DATASET, VERSION],
)

const { rows: cases } = await db.query(
  `SELECT id, prompt, reference FROM eval_case WHERE id = ANY($1::uuid[])`,
  [ds.case_ids],
)

for (const c of cases) {
  const t0 = Date.now()
  const res = await ai.messages.create({
    model: MODEL,
    max_tokens: 2048,
    system: await fs.readFile('prompts/system.md', 'utf8'),
    messages: [{ role: 'user', content: c.prompt }],
  })
  const out = res.content.map(b => b.type === 'text' ? b.text : '').join('')

  const { pass, score, reason } = c.reference
    ? exactMatch(out, c.reference)
    : await llmJudge(out, c.prompt)        // separate small-model call

  await db.query(
    `INSERT INTO eval_run
       (dataset_id, case_id, model, prompt_rev, output,
        pass, judge_score, judge_reason, latency_ms,
        input_tokens, output_tokens, cost_usd)
     VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12)`,
    [ds.id, c.id, MODEL, PROMPT_REV, out,
     pass, score, reason, Date.now() - t0,
     res.usage.input_tokens, res.usage.output_tokens, priceUsd(res.usage, MODEL)],
  )
}

There is no eval framework here. There is a for loop and an INSERT. That is the whole point — the eval is the SQL you write afterwards.

The eval is the SQL afterwards

This is what you actually wanted all along.

-- Pass rate per model on the latest core dataset
SELECT model, prompt_rev,
       count(*)            AS n,
       avg(pass::int)::numeric(4,3) AS pass_rate,
       avg(judge_score)::numeric(4,3) AS mean_score,
       sum(cost_usd)::numeric(8,2)   AS run_cost
FROM eval_run r
JOIN eval_dataset d ON d.id = r.dataset_id
WHERE d.name = 'core' AND d.version = 4
GROUP BY model, prompt_rev
ORDER BY pass_rate DESC;
-- Regressions: cases the new model fails that the old one passed
WITH old AS (
  SELECT case_id, pass FROM eval_run
  WHERE model = 'claude-opus-4-6' AND prompt_rev = 'a1b2c3'
), new AS (
  SELECT case_id, pass, output FROM eval_run
  WHERE model = 'claude-opus-4-7' AND prompt_rev = 'a1b2c3'
)
SELECT c.id, c.prompt, c.tags, new.output
FROM eval_case c
JOIN old ON old.case_id = c.id
JOIN new ON new.case_id = c.id
WHERE old.pass AND NOT new.pass
ORDER BY 'regression' = ANY(c.tags) DESC;
-- Coverage by tag — which areas of the dataset are thin?
SELECT unnest(tags) AS tag, count(*) AS cases
FROM eval_case
WHERE id = ANY((SELECT case_ids FROM eval_dataset WHERE name='core' AND version=4))
GROUP BY 1
ORDER BY 2 DESC;
-- "Find me cases like the one that just broke production"
SELECT id, prompt, tags
FROM eval_case
ORDER BY embedding <=> (SELECT embedding FROM eval_case WHERE id = $1)
LIMIT 10;

That last one is the killer feature CLI evals do not have. When a production failure lands, you embed the failing prompt and the database hands you the ten most-similar historical cases — your regression set writes itself.

When this is overkill

  • Solo prototype, two prompts. Stay in a notebook. You will outgrow it; you have not yet.
  • No judge signal. If neither exact-match nor an LLM judge can score the output, the table is just a log. Fix the judging problem before the storage problem.
  • Daily replay against a frontier model. Cost discipline matters more than capture discipline. Sample first, run nightly on a small slice, full runs only on model bumps and prompt-revision PRs.

The shape we end up with — cases are facts, datasets are snapshots, runs are joins — is the same shape every mature evaluation system has, from ML research benchmarks to compiler test suites. The novelty is just that the agent feeds itself: every interesting turn it ever has becomes a row that the next model release has to answer for.

#claude-code#agents#evals#regression-testing#data

© 2026 RedDB.io. AGPL-3.0 self-host · Managed Cloud.