workflows

Queues

Q: FIFO or priority?

Default is FIFO within a consumer group. `WITH PRIORITY` on the `CREATE QUEUE` statement enables a priority field on `PUSH` so high-priority jobs jump the line.

Q: Can I peek without acknowledging?

`QUEUE PEEK` returns jobs without removing them. Useful for dashboards and dry-runs.

FIFO, priority, consumer groups — durable workflow primitives.

Try Queues free See all 9 data models

What it is

Queues on RedDB.

Engine-native queue primitive with CREATE QUEUE, QUEUE PUSH / POP / PEEK, consumer groups, dead-letter queues, max-attempts retry policy.

Same WAL as your tables, so messages are durable without a separate RabbitMQ. A job that produces a row and enqueues a follow-up runs atomically in one transaction.

Pairs with the orchestrator pattern — long-lived worker calls consume(), processes, acks. Failed jobs land in the DLQ with the original payload preserved.

Code

Write it. Read it. Same engine.

Insert

QUEUE PUSH investigations {"case":"AB1234567","priority":"high"};

Query

QUEUE READ investigations GROUP forensics CONSUMER worker1 COUNT 10;

Driver examples

Connect from your language.

Real driver code — copy-paste ready, no pseudocode.

import { createQueue } from '@reddb/client'

const queue = createQueue('order_emails', { group: 'mailer-workers' })

// Long-lived worker — receives jobs one at a time
await queue.consume(async (job) => {
  try {
    await sendEmail(job.payload)
    return { ack: true }
  } catch (err) {
    // nack releases the job; after MAX_ATTEMPTS it goes to DLQ
    return { ack: false, reason: (err as Error).message }
  }
})

use reddb::Client;
use serde_json::json;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let client = Client::connect(std::env::var("REDDB_URL")?).await?;

    // Push a job — atomic with any other statement in the same transaction
    client
        .execute(
            "QUEUE PUSH order_emails $1",
            &[&json!({ "to": "alice@co.com", "template": "welcome" }).to_string()],
        )
        .await?;

    println!("job enqueued");
    Ok(())
}

Use cases

Where queues earn their place.

Background jobs

Email send, image resize, webhook fan-out — anywhere you would reach for Sidekiq or a Kafka topic.
Provisioning workflows

Multi-step orchestrations where each step writes a row and enqueues the next.
Webhook outbox

Atomic transactional outbox pattern — enqueue and update in one commit.
Retry + DLQ pipelines

Failed jobs surface in a dead-letter queue with their original payload for triage and replay.

Build it

End-to-end walkthrough.

Four steps from empty database to a real queues workload — each step shows the exact code and explains what the engine is doing.

Step 1

1. Create the queue (and a DLQ)

Two queues: the working queue and its dead-letter target. After 3 failed attempts a job is moved to the DLQ with its original payload + the failure reason — no manual replay loop, no message loss.
```
CREATE QUEUE IF NOT EXISTS provision_dlq;

CREATE QUEUE IF NOT EXISTS provision
  WITH DLQ provision_dlq
  MAX_ATTEMPTS 3;
```
Step 2

2. Enqueue inside a transaction

Transactional outbox without the library: the row insert and the enqueue commit together, or neither does. No "I wrote the row but the message lost in the broker" failure mode.
```
BEGIN;
  INSERT INTO databases (id, org_id, status)
  VALUES ('db_1', 'org_42', 'provisioning');

  QUEUE PUSH provision
    '{"database_id":"db_1","operation":"create"}';
COMMIT;
```

Step 3

3. Consume in a worker loop

A long-lived worker calls consume and gets jobs streamed to it. Acknowledge to remove the job; nack to release it for retry. After MAX_ATTEMPTS the job lands in the DLQ automatically.

// Worker (TypeScript)
const queue = createQueue('provision', { group: 'workers' })

await queue.consume(async (job) => {
  try {
    await provisionDatabase(job.payload)
    return { ack: true }
  } catch (err) {
    return { ack: false, reason: err.message }
    // job goes back on the queue with attempts++
  }
})

Step 4

4. Drain + replay the DLQ

DLQ is inspectable with QUEUE PEEK. After triaging by error pattern and fixing the upstream issue, re-push the corrected job to the live queue and ACK it off the DLQ inside a transaction so the two moves are atomic.

-- See what is stuck.
QUEUE PEEK provision_dlq 50;

-- Fix the upstream issue, then re-push each job and ACK it off the DLQ.
-- (app code iterates the PEEK results)
BEGIN;
  QUEUE PUSH provision '{"database_id":"db_1","operation":"create"}';
  QUEUE ACK provision_dlq <msg_id>;
COMMIT;

Engine features

What RedDB adds on top of queues.

The engine knobs you would otherwise wire up yourself — TTL, encryption, indexes, atomic ops, eventual consistency — applied to this data model.

Dead-letter queue (DLQ)

CREATE QUEUE … WITH DLQ <name> MAX_ATTEMPTS N declares a target for jobs that exhaust retries. Failed payloads + last error land in the DLQ as queryable rows — no message loss, no custom replay script.
Consumer groups

A queue can have many consumer groups; within one group, jobs are partitioned across the consumers. Add workers to scale throughput linearly until the engine becomes the bottleneck.
ACID transactional outbox

BEGIN; INSERT row; QUEUE PUSH; COMMIT; either fully lands or fully rolls back. Replaces the entire transactional-outbox library category for application workflows.
FIFO + priority

Default is FIFO within a consumer group. WITH PRIORITY on the queue declaration enables a priority field on PUSH so high-priority jobs jump the line.
PEEK without consume

QUEUE PEEK reads jobs without removing them — useful for dashboards, dry-runs, debugging.
DLQ replay

QUEUE PEEK dlq N inspects stuck jobs without removing them. After fixing the upstream issue, re-push to the live queue and QUEUE ACK on the DLQ inside a transaction — atomic, no custom replay script needed.
Same WAL / fsync as table writes

Every PUSH is durable through the same WAL + fsync path that table writes use. Crash recovery preserves every committed message — no separate broker durability story to maintain.
Per-queue policies

A worker role can consume from provision but not consume from billing. Producers gated separately from consumers. Policy attachment surfaces in audit log alongside the message lineage.
Audit log of every state transition

Every PUSH / POP / ACK / NACK / DLQ-move is logged as an audit row — actor, message id, transition, timestamp. Forensics on a stuck workflow becomes a SELECT, not a journey through 3 broker dashboards.
PITR via WAL archive

Replay the queue state to a wall-clock timestamp. Useful for "what was in flight when the bad deploy went out" reconstructions; pair with the audit log for a full timeline.

Showcases

What queues can do once you bring the rest of the engine in.

Cross-model correlation, algorithm flex, multi-model patterns — things that take a stack of services elsewhere.

Saga orchestrator with table-backed state

Multi-step provisioning workflow whose state lives in a regular table — queue is the trigger.

A "create database" saga has 4 steps: allocate volume → start machine → run migrations → email user. Each step writes the state row + enqueues the next step in one transaction. If anything fails, the row stays at the last successful step; a retry consumer picks it up where it left off.

sql

BEGIN;
  UPDATE provisioning
     SET state = 'volume_allocated',
         volume_id = $1
   WHERE database_id = $2;

  QUEUE PUSH provision_step
    json_build_object(
      'database_id', $2,
      'next_step',  'start_machine'
    );
COMMIT;
-- Volume allocation + state transition + next-step trigger
-- are atomic. If COMMIT fails, the saga has not advanced.

Cross-type observability: queue depth as a time-series

Worker writes consume metrics into the time-series collection — queue health visible in your existing dashboards.

On every consume / ack / nack, the worker records a metric point. Queue depth, error rate, and p99 processing time become time_bucket queries — same dashboard as everything else, no broker-specific Prometheus exporter to maintain.

typescript

await q.consume(async (job) => {
  const start = Date.now()
  try {
    await processJob(job.payload)
    await db.timeseries.insert('queue_metrics', [{
      timestamp: Date.now(),
      metric: 'job.duration_ms',
      value: Date.now() - start,
      tags: { queue: 'provision', status: 'ok' },
    }])
    return { ack: true }
  } catch (err) {
    await db.timeseries.insert('queue_metrics', [{
      timestamp: Date.now(),
      metric: 'job.duration_ms',
      value: Date.now() - start,
      tags: { queue: 'provision', status: 'error' },
    }])
    return { ack: false, reason: err.message }
  }
})

Migrate from

Bring queues over from your current tool.

From

RabbitMQ

Drain the RabbitMQ queue, then re-enqueue against RedDB. Worker code swap is one connection string + driver call.

# Drain (per queue you migrate)
rabbitmqadmin get queue=invoicing count=10000 \
  | jq -c '.[] | { payload: .payload }' > invoicing.jsonl

# Replay
red sql -d "reds://admin@.../my-db:5050" \
  --file <(awk '{print "QUEUE PUSH invoicing \u0027" $0 "\u0027;"}' invoicing.jsonl)

From

AWS SQS

Use the AWS CLI to receive-and-delete in batches, replay each batch as a transactional QUEUE PUSH on RedDB.

aws sqs receive-message --queue-url $URL --max-number-of-messages 10 \
  | jq -c '.Messages[] | .Body' \
  | while read body; do
      red sql -d "reds://admin@.../my-db:5050" \
        "QUEUE PUSH provision '$body';"
    done

Recipes

Common patterns, real driver code.

sql

Transactional outbox

Append a side-effect message in the same commit as the primary row. No "I wrote the row but Kafka was down" failure mode.

BEGIN;
  INSERT INTO orders (id, customer_id, total)
  VALUES ($id, $customer_id, $total);

  QUEUE PUSH order_emails
    json_build_object('order_id', $id, 'customer_id', $customer_id);
COMMIT;

typescript

Long-running consumer with structured retries

Worker calls consume(), processes, acks. Retries surface in the DLQ after MAX_ATTEMPTS without a custom retry library.

import { createQueue } from '@reddb-io/client'

const q = createQueue('order_emails', { group: 'mailer' })

await q.consume(async (job) => {
  try {
    await sendOrderEmail(job.payload)
    return { ack: true }
  } catch (err) {
    return { ack: false, reason: err.message }
    // attempt++; on attempt 4, jumps to order_emails_dlq.
  }
})

sql

DLQ triage and replay

Inspect failed jobs with QUEUE PEEK, then re-push and ACK inside a transaction to move them back to the live queue.

-- triage: inspect without consuming
QUEUE PEEK order_emails_dlq 20;

-- replay after the fix: re-push + ACK atomically
BEGIN;
  QUEUE PUSH order_emails '{"order_id":"ord_99","customer_id":"cust_7"}';
  QUEUE ACK order_emails_dlq <msg_id>;
COMMIT;

Limits

What it does. What it costs.

Operational caps and durability semantics, no hand-waving.

Throughput envelope: ~50k msg/sec sustained per queue (commodity hw)
Max payload size: 1 MB per message
Retry policy: Per-queue MAX_ATTEMPTS + DLQ — no extra libs
Consumer groups: Many groups per queue, partitioned within each group
Durability: fsync + WAL on every PUSH, same as table writes

When NOT to use

Don't pick queues for these.

Honest constraints — when another model fits better.

Don't use Queues for high-throughput event streams

Above ~50k msg/sec sustained, Kafka or NATS are sized for the workload. RedDB queues are application-tier — durable orchestration, not event-sourcing fan-out.
Don't keep messages on the queue for indefinite history

A queue is a work item, not a log. If you need replay-from-genesis semantics, write the events to a table and tail with cursor pagination.
Avoid acking before the side effect commits

Process inside a transaction or with idempotency keys. An ack means "this work is done"; it should be honest about that, not optimistic.

How RedDB queues compare.

RabbitMQ / SQS

Atomic with the rest of your data. Enqueue + update in one transaction, no transactional outbox library required.

Kafka

Different scale envelope — RedDB queues fit application-tier workflows, not 1M-msg/sec event streams.

Pairs with

Queues earns more next to these.

The data models queues compose with — a click away.

FAQ

Questions teams ask before they pick queues.

FIFO or priority?

Default is FIFO within a consumer group. WITH PRIORITY on the CREATE QUEUE statement enables a priority field on PUSH so high-priority jobs jump the line.

Are messages durable?

Yes. Every PUSH goes through the WAL with the same fsync guarantees as a row write. Crash recovery preserves every committed message.

How many consumers can read the same queue?

A queue can have many consumer groups; within one group, jobs are partitioned across the consumers. Add workers to scale throughput linearly until the engine becomes the bottleneck.

When should I NOT use this — Kafka territory?

Above ~50k messages/second sustained, or when you need indefinite retention of every event for replay (event sourcing). RedDB queues are sized for application workflows, not high-throughput event streams.

Can I peek without acknowledging?

QUEUE PEEK returns jobs without removing them. Useful for dashboards and dry-runs.

Explore

Every other data model on RedDB.

One engine, 9 first-class data models. Mix as needed.

Try queues on RedDB.

Free nano database, no credit card. AGPL self-host or managed Cloud — same engine, same file format.

Start free See pricing

Queues

Queues on RedDB.

Write it. Read it. Same engine.

Connect from your language.

Where queues earn their place.

Background jobs

Provisioning workflows

Webhook outbox

Retry + DLQ pipelines

End-to-end walkthrough.

1. Create the queue (and a DLQ)

2. Enqueue inside a transaction

3. Consume in a worker loop

4. Drain + replay the DLQ

What RedDB adds on top of queues.

What queues can do once you bring the rest of the engine in.

Saga orchestrator with table-backed state

Cross-type observability: queue depth as a time-series

Bring queues over from your current tool.

RabbitMQ

AWS SQS

Common patterns, real driver code.

Transactional outbox

Long-running consumer with structured retries

DLQ triage and replay

What it does. What it costs.

Don't pick queues for these.

Don't use Queues for high-throughput event streams

Don't keep messages on the queue for indefinite history

Avoid acking before the side effect commits

How RedDB queues compare.

Queues earns more next to these.

Tables →

Time-series →

Key-Value →

Questions teams ask before they pick queues.

FIFO or priority?

Are messages durable?

How many consumers can read the same queue?

When should I NOT use this — Kafka territory?

Can I peek without acknowledging?

Every other data model on RedDB.

Tables →

Documents →

Graphs →

Vectors →

Key-Value →

Cache →

Time-series →

Probabilistic →

Try queues on RedDB.

Tables

Time-series

Key-Value

Tables

Documents

Graphs

Vectors

Key-Value

Cache

Time-series

Probabilistic