Agents · 2026-06-03 · By RedDB team · 4 min read

Sub-agent dispatch via database queue — durable parallelism across sessions

Claude Code's Task tool spawns sub-agents that die with the session. Push the work into a RedDB queue instead and workers on any machine pick it up, write results back, and the parent subscribes to completions via LISTEN/NOTIFY. Fire-and-forget becomes durable fan-out.

The Task tool’s shelf life is one session

Task(subagent_type=…) is the most underused tool in Claude Code. Spawn five sub-agents, each chews on an independent slice, parent collects results — except every one of those sub-agents lives inside the parent’s process. Close the laptop, kill the terminal, hit a context-window cap on the parent, and the work evaporates. Mid-flight tool calls are simply gone; their partial output is unrecoverable.

That’s fine for “explore the codebase, summarize” — disposable. It’s not fine for “run the eval suite against 200 prompts,” “lint and auto-fix 80 packages,” “regenerate embeddings for every doc touched this week.” Those are the jobs you actually want to send AFK. They need to survive a restart.

The pattern in this post: push sub-agent tasks into a RedDB queue, let one or more workers pull from it, write results back to the same row, and have the parent session subscribe to completions via LISTEN/NOTIFY. Same Task ergonomics on the surface, durable fan-out underneath. Builds on the memory backend from D1, the transactional-hook pattern from D3, and the observability schema from D9.

The schema: one table, three states, a notify trigger

CREATE TYPE task_state AS ENUM ('pending', 'running', 'done', 'failed');

CREATE TABLE agent_task (
  id           uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  parent_id    text NOT NULL,                       -- session id of dispatcher
  kind         text NOT NULL,                       -- 'review', 'embed', 'eval'…
  payload      jsonb NOT NULL,                      -- prompt, file list, cfg
  state        task_state NOT NULL DEFAULT 'pending',
  worker_id    text,
  attempts     int NOT NULL DEFAULT 0,
  max_attempts int NOT NULL DEFAULT 3,
  result       jsonb,
  error        text,
  idem_key     text UNIQUE,                         -- dedupe on retry
  created_at   timestamptz NOT NULL DEFAULT now(),
  started_at   timestamptz,
  finished_at  timestamptz,
  visible_at   timestamptz NOT NULL DEFAULT now()   -- delayed retry, lease
);

CREATE INDEX agent_task_claim_idx
  ON agent_task (visible_at)
  WHERE state = 'pending';

CREATE INDEX agent_task_parent_idx ON agent_task (parent_id, state);

CREATE OR REPLACE FUNCTION notify_task_done() RETURNS trigger AS $$
BEGIN
  IF NEW.state IN ('done','failed') AND OLD.state <> NEW.state THEN
    PERFORM pg_notify('task_done_' || NEW.parent_id, NEW.id::text);
  END IF;
  RETURN NEW;
END $$ LANGUAGE plpgsql;

CREATE TRIGGER agent_task_notify
  AFTER UPDATE ON agent_task
  FOR EACH ROW EXECUTE FUNCTION notify_task_done();

Four moving parts worth a comment. idem_key lets a flaky dispatcher retry without double-queuing — same key, second insert returns the existing row. visible_at doubles as both delayed retry (push it ten seconds into the future after a failure) and worker lease (a claimer bumps it 5 minutes out so a peer doesn’t steal in-flight work). The partial index on (visible_at) WHERE state = 'pending' keeps the claim query a single index range scan even at a million rows. The pg_notify channel is per-parent so a dispatcher hears only its own completions — no fan-out storm to unrelated sessions.

Dispatcher: a tiny wrapper around Task

// .claude/lib/dispatch.ts
import { Pool } from 'pg'
const pool = new Pool({ connectionString: process.env.REDDB_URL })

export async function dispatch(kind: string, payload: unknown, opts: {
  parentId: string
  idemKey?: string
  maxAttempts?: number
}) {
  const { rows } = await pool.query(
    `INSERT INTO agent_task (parent_id, kind, payload, idem_key, max_attempts)
     VALUES ($1, $2, $3, $4, COALESCE($5, 3))
     ON CONFLICT (idem_key) DO UPDATE SET idem_key = EXCLUDED.idem_key
     RETURNING id`,
    [opts.parentId, kind, payload, opts.idemKey ?? null, opts.maxAttempts ?? null],
  )
  return rows[0].id as string
}

export async function awaitAll(parentId: string, ids: string[], timeoutMs = 600_000) {
  const client = await pool.connect()
  try {
    await client.query(`LISTEN "task_done_${parentId}"`)
    const pending = new Set(ids)
    const results = new Map<string, any>()

    const drain = async () => {
      const { rows } = await client.query(
        `SELECT id, state, result, error FROM agent_task
          WHERE parent_id = $1 AND id = ANY($2::uuid[]) AND state IN ('done','failed')`,
        [parentId, [...pending]],
      )
      for (const r of rows) { results.set(r.id, r); pending.delete(r.id) }
    }
    await drain()

    if (pending.size === 0) return results
    return await new Promise<typeof results>((resolve, reject) => {
      const timer = setTimeout(() => reject(new Error('dispatch timeout')), timeoutMs)
      client.on('notification', async (msg) => {
        if (!pending.has(msg.payload!)) return
        await drain()
        if (pending.size === 0) { clearTimeout(timer); resolve(results) }
      })
    })
  } finally {
    client.release()
  }
}

The ON CONFLICT … DO UPDATE … RETURNING id dance is the idempotent-insert trick — same idem_key returns the original row’s id, so a dispatcher that crashes between two INSERTs and resumes won’t enqueue twice. awaitAll does a drain-then-listen so completions that landed before the LISTEN registered still get picked up; without the initial drain you’d have a race where fast workers finish before the parent subscribes.

Worker: claim, run, write back

// workers/agent-worker.ts — run one per machine, or one per CPU core
import { Pool } from 'pg'
import { runSubagent } from './runner'   // your wrapper around the Claude API or CLI
const pool = new Pool({ connectionString: process.env.REDDB_URL })
const workerId = `${process.env.HOSTNAME}-${process.pid}`

async function claim() {
  const { rows } = await pool.query(`
    UPDATE agent_task
       SET state = 'running',
           worker_id = $1,
           started_at = now(),
           visible_at = now() + interval '5 minutes',
           attempts = attempts + 1
     WHERE id = (
       SELECT id FROM agent_task
        WHERE state = 'pending' AND visible_at <= now()
        ORDER BY created_at
        FOR UPDATE SKIP LOCKED
        LIMIT 1
     )
     RETURNING id, kind, payload, attempts, max_attempts
  `, [workerId])
  return rows[0]
}

async function finish(id: string, ok: boolean, body: unknown) {
  await pool.query(
    `UPDATE agent_task
        SET state = $2, ${ok ? 'result' : 'error'} = $3, finished_at = now()
      WHERE id = $1`,
    [id, ok ? 'done' : 'failed', ok ? body : String(body)],
  )
}

async function retry(id: string, attempts: number, max: number, err: unknown) {
  if (attempts >= max) return finish(id, false, err)
  const backoff = Math.min(60, 2 ** attempts) // seconds
  await pool.query(
    `UPDATE agent_task
        SET state = 'pending', visible_at = now() + ($2 || ' seconds')::interval, error = $3
      WHERE id = $1`,
    [id, backoff, String(err)],
  )
}

while (true) {
  const task = await claim()
  if (!task) { await new Promise(r => setTimeout(r, 1000)); continue }
  try {
    const out = await runSubagent(task.kind, task.payload)
    await finish(task.id, true, out)
  } catch (err) {
    await retry(task.id, task.attempts, task.max_attempts, err)
  }
}

FOR UPDATE SKIP LOCKED is the load-bearing line. Drop it and N workers serialize on the same row lock; keep it and they fan out cleanly across the queue. The visibility-timeout update inside the same statement is the lease — if this worker crashes mid-task, the row reverts to claimable after five minutes because a watchdog (next section) flips state back to pending. The exponential backoff on retry tops out at one minute so failed tasks don’t park in the queue for hours.

Watchdog: reclaim leases, alert on poison pills

-- run every minute from cron or pg_cron
UPDATE agent_task
   SET state = 'pending', visible_at = now() + interval '5 seconds'
 WHERE state = 'running'
   AND started_at < now() - interval '5 minutes';

-- alert: tasks that hit max_attempts
SELECT id, kind, error, attempts
  FROM agent_task
 WHERE state = 'failed'
   AND finished_at > now() - interval '1 hour';

First query is the lease reclaimer. Second is the canary — wire it to the observability dashboard from D9 and a Slack webhook so poison pills surface within an hour instead of clogging the queue silently.

Calling it from a Claude Code session

// .claude/skills/parallel-review/run.ts
import { dispatch, awaitAll } from '../../lib/dispatch'

const parentId = process.env.CLAUDE_SESSION_ID!
const files: string[] = JSON.parse(process.env.CHANGED_FILES!)

const ids = await Promise.all(files.map(f =>
  dispatch('review', { file: f, ruleset: 'house-style' }, {
    parentId,
    idemKey: `${parentId}:${f}`,
  })
))

const results = await awaitAll(parentId, ids, 15 * 60_000)
for (const [, r] of results) {
  console.log(r.state === 'done' ? r.result : `FAIL: ${r.error}`)
}

From the model’s perspective this looks like a synchronous fan-out, but the rows live in the DB. Kill the session, restart it tomorrow, re-run with the same parentId and idem_keys, and awaitAll returns immediately for tasks already finished and waits only on the stragglers. That’s the durable part.

When this pattern earns its keep

Three signals that you’ve outgrown raw Task:

  1. Single fan-out > 30 sub-agents. In-process parallelism starts to choke on rate limits and the parent’s context fills with sub-agent output it has to scroll past.
  2. Work outlasts a session. Anything where “let it run overnight” is a normal answer — evals, batch refactors, doc regeneration.
  3. More than one machine has spare cycles. A laptop and a workstation can both pull from the same queue; the queue is the only piece of shared state.

If none of those hold, Task is still the right primitive — don’t drag in a database to dispatch three sibling agents that finish in 90 seconds.

What’s next

D13 (eval datasets) builds on this queue: every dispatched eval becomes a row, every replay against a new model version is another fan-out against the same set of payloads. D14 (cross-session task queues) generalises the worker loop into the AFK-agent runtime: same table, same claim query, plus checkpoints and resume tokens so a sub-agent can be paused mid-thought and revived on a different host.

#claude-code#agents#multi-agent#queues#durability

© 2026 RedDB.io. AGPL-3.0 self-host · Managed Cloud.