Disaster recovery for RedDB: PITR, snapshots, regional failover

This is the post we hand to operators after the self-host checklist. The checklist gets you running. This one gets you back up when something goes wrong — and tells you, with numbers, how long that takes.

DR posts tend to oversell. We won’t. RedDB gives you good RPO and decent RTO on a single region, and asynchronous cross-region replication that is honest about its lag. It does not give you synchronous multi-region writes. If you need RPO=0 across continents, RedDB is the wrong answer today, and the non-applicability post says so out loud.

The three failures you plan for

Most operator panic falls into one of three shapes, and your DR strategy has to cover each one differently:

Logical corruption — a bad migration, an UPDATE without a WHERE, a poisoned write from a regression. You want to rewind to just before the bad thing happened. PITR is the answer.
Volume loss — the underlying disk is gone (rare on managed storage, possible on bare metal). The cluster is otherwise healthy elsewhere. Snapshot + WAL replay is the answer.
Region loss — the whole AZ or region is gone or unreachable for hours. Cross-region replica promotion is the answer.

These three call for different artifacts (continuous WAL archive, periodic full snapshot, follower replica) and different RPO/RTO targets. Sizing all three correctly is most of the work.

Point-in-time restore (logical corruption)

Logical corruption is the most common DR scenario in practice. It is also the one where naive “we have backups” answers fail — a nightly snapshot does not help if the bad write happened at 14:30 and the next snapshot is at 02:00.

RedDB’s PITR machinery is two pieces:

Base snapshot — a consistent full copy taken on a schedule (default: daily at 02:00 UTC, before the EU traffic ramp).
Continuous WAL archive — every WAL segment is shipped to object storage as it seals, typically every 16 MiB or every 30 seconds (whichever comes first).

To restore to an arbitrary point in time, you fetch the most recent snapshot before the target time, then replay the archived WAL forward up to the requested timestamp.

# Restore to 2026-04-28 14:29:00 UTC (one minute before the bad UPDATE)
reddbctl restore \
  --target-time '2026-04-28T14:29:00Z' \
  --snapshot-bucket s3://reddb-backups-prod \
  --wal-bucket s3://reddb-wal-prod \
  --destination /var/lib/reddb-restore \
  --master-key-arn arn:aws:kms:us-east-1:1234:key/abcd-ef

The command fetches the snapshot taken at 02:00 the same day (~12 hours of WAL to replay), unwraps the KMS-encrypted at-rest key, applies the WAL segments in order, and stops at the timestamp you asked for.

In practice you do not restore in place over a live cluster. You restore to a parallel data directory, point a single read-only RedDB process at it, run your forensic queries to confirm the rewind landed on the right side of the bad write, then decide whether to swap the live cluster or surgically extract rows back into it. The second option (surgical extraction) is almost always the right one — full-cluster rewind discards every legitimate write between the bad event and now.

RPO for PITR: bounded by your WAL ship interval. Default is 30 seconds, meaning the worst case is 30 seconds of writes lost if the region itself dies between WAL ships. For an in-region logical corruption restore (the common case), RPO is effectively zero — the WAL has already been shipped before you decide to restore.

RTO for PITR: bounded by snapshot size and WAL replay rate. On a 200 GiB dataset with one day of WAL to replay, expect 20–40 minutes wall-clock. Most of that is the snapshot download, not the replay.

Snapshot scheduling

The default daily-at-02:00 snapshot is fine for most teams. The two cases where you change it:

You have a strict RTO and 12+ hours of WAL replay would blow it. Snapshot twice a day, or every 6 hours. Cost scales linearly with snapshot frequency — full snapshots are expensive on large datasets.
Your write pattern is bursty and a 24-hour replay would include the daily batch job that writes 100 GiB. Snapshot after the batch job, not before.

The G-27 checklist post showed the CronJob manifest. Here is the schedule decision:

# values.yaml — snapshot schedule decision points
snapshots:
  schedule: "0 2 * * *"              # daily 02:00 UTC
  retention_days: 35                  # > 30-day audit window + buffer
  encryption_key_arn: ${KMS_KEY_ARN}  # always present, never optional
  destination: s3://reddb-backups-prod
  # If your RTO is < 4 hours, uncomment and run twice daily:
  # schedule: "0 2,14 * * *"

Two retention rules:

Snapshots: keep ≥ (your max RTO window) + 7 days of full snapshots. The +7 covers “we discovered the corruption two days late.”
WAL: keep ≥ 35 days of WAL archive even though that’s longer than most snapshot retention. The reason: an auditor or legal hold sometimes asks for “data as it existed on date X,” and WAL replay is the only way to answer that without keeping every snapshot forever.

Both buckets must be in a different failure domain than the live cluster — different region or, at minimum, a different storage account with separate IAM. The most common DR mistake we see is backups stored in the same bucket as the cluster’s working data. When the cluster’s bucket dies, so do the backups.

Cross-region replication topology

RedDB ships an asynchronous replication mode where one or more follower clusters in other regions stream the WAL from the leader and apply it. This gives you:

A warm-standby cluster you can promote if the leader region dies.
A read-only replica in another region for read-locality (a useful side benefit).
A continuously-tested DR target — if replication is healthy, you know the recovery path works right now, not “in theory next quarter.”

The topology we recommend for most teams is one leader and one async follower, both behind their own load balancer, with DNS pointed at the leader by default.

┌─────────────────┐         WAL stream         ┌─────────────────┐
│ Leader (us-east)│ ────────────────────────▶  │ Follower (eu-w) │
│   r/w traffic   │                            │   read-only     │
└────────┬────────┘                            └────────┬────────┘
         │                                              │
         ▼                                              ▼
   app traffic                                  read-only fan-out
       (default)                              (optional, lower-priority)

The replication lag metric (replication_lag_bytes and replication_lag_seconds) is one of the four metrics the G-27 checklist said to alert on. For most workloads, expect 0.5–3 seconds of lag, with bursts to ~30 seconds during compaction. If you see sustained lag > 60 seconds, page someone — the follower is no longer a viable failover target until it catches up.

Promoting a follower

When the leader region is gone (or sufficiently broken that you’ve made the call to fail over), promote the follower:

# On the follower cluster:
reddbctl promote \
  --confirm-leader-down \
  --new-leader-region eu-west-1

The --confirm-leader-down flag is intentional friction. We have seen exactly one customer accidentally promote a follower while the leader was still serving writes; the resulting split-brain took six hours to clean up. The flag forces an operator to acknowledge they have already verified the leader is unreachable from the application tier and from the follower itself.

After promotion, your DNS or service-mesh routing has to swap. We do not automate the DNS swap. Most teams’ DNS automation is bespoke and brittle, and an automatic regional failover that fires on a transient network blip is worse than the disaster it was supposed to mitigate. Manual swap, executed by a human who has read a runbook, is the right tradeoff for almost everyone.

RPO for regional failover: equal to replication lag at the moment of leader loss. If lag was 2 seconds, you lose 2 seconds of writes. If lag was 60 seconds, you lose 60 seconds.

RTO for regional failover: dominated by DNS propagation + connection-pool reset in the application tier. Cluster-side promotion is sub-30s. Realistic end-to-end RTO is 5–15 minutes depending on how aggressive your DNS TTLs are and whether your application has connection retries that survive a backend swap.

The numbers you can hit

For a midsize deployment (200 GiB dataset, 5K writes/sec, daily snapshot, 30s WAL ship interval, one async cross-region follower):

Failure	RPO	RTO	Tooling
Logical corruption	~0 (in-region)	20–40 min	`reddbctl restore --target-time`
Volume loss	< 30 sec	15–25 min	Snapshot + WAL replay onto new PV
Region loss	2–30 sec	5–15 min	`reddbctl promote` + DNS swap
Whole-account loss	1 day	4+ hours	Restore from cross-account vault

These are not aspirations. They are the numbers we measure on our own deployments. Your numbers will differ — bigger datasets push RTO up roughly linearly, faster networks pull it down, and applications with aggressive connection retries shave 1–2 minutes off the failover number.

If your business needs better than these, the answer is not to tune RedDB harder. The honest answer is below.

What we don’t yet do

DR sales decks lie. Here is what RedDB explicitly does not provide today, and what you should pick instead if you need it:

Synchronous multi-region writes — no. Writes commit on the leader before being replicated. If you need RPO=0 across regions, look at Spanner, CockroachDB, or YugabyteDB. They pay a meaningful latency cost on the write path; that cost is unavoidable for the property you’re buying.
Automatic regional failover — no. The promote command is manual on purpose. See above.
Cross-region read-after-write consistency — no. The follower is eventually consistent. If your application reads through the EU follower and expects writes from the US leader to be visible immediately, you will see stale reads. Pin reads to the leader for that workload, or accept the lag.
In-cluster geo-distribution of a single dataset — no. Each cluster is regional. The follower is a replica, not a shard. If you need data residency by tenant (some EU customers, some US), run separate clusters per region and route by tenant ID at the application layer.
Backup verification automation — partial. We ship a reddbctl restore --dry-run that verifies snapshot and WAL integrity without writing to disk. We do not automatically restore your nightly snapshot into a scratch cluster every day. That drill remains a human responsibility, and it is item #8 on the self-host checklist for a reason.

This list will shrink. The order matters: synchronous multi-region writes are not on the roadmap (the latency cost is not what most of our users want); cross-region read-after-write is on the roadmap (via causal tokens) and is the closest item to landing.

The drill you owe yourself

Pick a Tuesday. Set a timer. Pretend the leader region is gone.

Page yourself the way the real alert would page you.
Read your own runbook end-to-end. (You have a runbook, right?)
Promote the follower.
Swap DNS.
Verify the application is serving from the new region.
Read at least one row that was written less than 60 seconds before “the leader died.” Verify it’s there. Note what was lost if anything.
Restore the leader region. Decide whether to fail back or stay on the new leader.

The first time a team runs this drill, it takes 90 minutes longer than they estimated, they discover three runbook errors, and the application’s connection-pool retry logic is wrong in some non-obvious way. That’s the entire point of running it before the real outage. Schedule the next one for 90 days out.

TL;DR

Three failure shapes (logical corruption, volume loss, region loss) need three different DR artifacts. Cover all three or you don’t have DR; you have hope.
PITR via base snapshot + continuous WAL archive: RPO ~zero for in-region restores, RTO 20–40 minutes on a midsize dataset.
Async cross-region follower: RPO 2–30 seconds, RTO 5–15 minutes including DNS swap.
Promotion is manual on purpose. Automatic regional failover causes more outages than it prevents.
No synchronous multi-region writes. If you need RPO=0 across continents, pick a different database.
Run the drill quarterly. The first drill always finds three real bugs.