Self-hosting RedDB: the real-world checklist

This is the post we wish existed when we first put RedDB on a customer’s cluster. It is not a marketing surface — it is the literal checklist our SRE walks through before declaring an environment “production-ready.” If you can answer every item below with “yes, and here’s the evidence,” you have a self-host that will not page you at 03:00.

Everything here assumes Kubernetes because that is where ~90% of self-hosters land. The same shape applies on plain VMs with systemd or on Nomad; the artifacts change, the decisions do not.

The five-minute version

Run this checklist before you put traffic on the cluster:

StatefulSet, not Deployment, for the data plane.
PersistentVolumeClaim sized at 3× expected hot working set, on SSD-class storage with fsync honored.
readinessProbe that checks the WAL is fsynced and a follower has caught up to within 5 seconds.
Prometheus scrape against /metrics on the admin port (not the data port).
A CronJob that runs a backup against object storage, with retention >= compliance window + 7 days.
NetworkPolicy that only allows the application namespace, the monitoring namespace, and the backup job to reach the data port.
KMS-backed secret for the encryption-at-rest key (see vault key rotation for why this matters).
A documented restore drill that someone ran in the last 90 days.

The rest of this post is the long form of those eight bullets.

Container image: pin, don’t `:latest`

image: registry.reddb.io/reddb:1.8.4
imagePullPolicy: IfNotPresent

Two reasons we belabor this. First, :latest defeats the whole point of immutable infra — a node reschedule silently rolls you forward. Second, our minor versions occasionally change defaults (compaction priority, bloom sizing) and you want the choice to be deliberate. Pin the patch, read the changelog before you bump the minor.

Image size is ~80 MiB compressed. We ship a -debug variant with pprof and a shell; do not use it as your default runtime — it has a strictly larger attack surface for no operational benefit.

StatefulSet, not Deployment

The data plane is stateful. The right Kubernetes primitive is StatefulSet. A working manifest looks like:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: reddb
  namespace: reddb-system
spec:
  serviceName: reddb-headless
  replicas: 3
  podManagementPolicy: Parallel
  selector:
    matchLabels:
      app: reddb
  template:
    metadata:
      labels:
        app: reddb
    spec:
      terminationGracePeriodSeconds: 120
      securityContext:
        runAsUser: 10001
        runAsGroup: 10001
        fsGroup: 10001
      containers:
        - name: reddb
          image: registry.reddb.io/reddb:1.8.4
          args: ["--config=/etc/reddb/config.toml"]
          ports:
            - name: data
              containerPort: 5432
            - name: replica
              containerPort: 5433
            - name: admin
              containerPort: 9180
          env:
            - name: REDDB_NODE_ID
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: REDDB_KMS_KEY_ARN
              valueFrom:
                secretKeyRef:
                  name: reddb-kms
                  key: arn
          volumeMounts:
            - name: data
              mountPath: /var/lib/reddb
            - name: config
              mountPath: /etc/reddb
              readOnly: true
          resources:
            requests:
              cpu: "2"
              memory: 8Gi
            limits:
              memory: 12Gi
          readinessProbe:
            httpGet:
              path: /readyz
              port: admin
            periodSeconds: 5
            failureThreshold: 3
          livenessProbe:
            httpGet:
              path: /livez
              port: admin
            periodSeconds: 30
            failureThreshold: 3
            initialDelaySeconds: 60
      volumes:
        - name: config
          configMap:
            name: reddb-config
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: gp3-fsync-honored
        resources:
          requests:
            storage: 200Gi

A few decisions inside this we want to call out, because the defaults regularly trip people:

podManagementPolicy: Parallel. The default is OrderedReady, which serializes startup. RedDB nodes form a quorum on boot — they need to come up roughly together, not one after the other waiting for the previous to be Ready.
terminationGracePeriodSeconds: 120. The data plane needs time to flush the WAL, hand off the leader role if it owned one, and close client connections cleanly. 30 seconds (the default) will cause WAL replay on restart, which adds startup latency and shows up as a recurring availability dip.
CPU request 2, no limit. Compaction is bursty. Setting a CPU limit will cause throttling during compaction windows and you will see the exact latency spike that the LSM compaction notes describe. Memory limits, however, are fine and recommended — RedDB respects the cgroup memory limit and shrinks its block cache accordingly.
livenessProbe.initialDelaySeconds: 60. WAL replay on a cold node can take a minute. Without the delay, a slow-starting node gets killed mid-replay, replays again, gets killed again. We have seen production clusters stuck in this loop for hours.

Storage: `gp3-fsync-honored`

This is the single line item that decides whether your self-host is durable or theatrical. RedDB’s durability guarantees assume fsync actually persists data before returning. Many cloud storage classes lie about this for performance. The classes that do not lie, by cloud:

AWS: gp3 (set iopsPerGB to at least 3, and confirm the EBS-CSI driver is not running with dataPlaneOptions.fsyncDisabled).
GCP: pd-ssd (the default pd-standard does honor fsync but the IOPS profile is unsuitable).
Azure: Premium SSD v2 (the older Premium_LRS works but costs more for less).
Bare metal: NVMe with LITTLE_ENDIAN write barriers enabled in the filesystem (ext4 with data=ordered, the default, is correct; data=writeback is not).

Size the PVC at 3× expected hot working set. The 3× accounts for: 1× the live data, ~0.5× the SSTables waiting on compaction, ~0.5× temporary space for compaction itself, and the rest as headroom for pgbench-style load tests and the occasional bloat. Resizing PVCs is supported but disruptive; oversize on day one.

Monitoring: the four metrics that matter

RedDB exports ~120 Prometheus metrics. Watch these four; alert on the first three:

Metric	What it tells you	Page threshold
`reddb_wal_fsync_seconds{quantile="0.99"}`	Storage subsystem health	> 50ms
`reddb_replication_lag_bytes{follower=~".*"}`	Follower divergence	> 100 MiB sustained 5min
`reddb_compaction_score{level=~".*"}`	Compaction debt	> 50 sustained 10min
`reddb_block_cache_hit_ratio`	Working set fits	(track, don’t page)

If wal_fsync_seconds spikes, your storage is the problem, full stop. Do not start tuning RedDB until that number is back under 10ms. We have spent days helping customers tune things that were not the bottleneck because they were watching the wrong dashboard.

If compaction_score is climbing without bound, you are writing faster than the cluster can compact. The fix is either more storage IOPS, more CPU, or write backpressure on the application side. RedDB has a built-in backpressure mechanism that returns SQLSTATE 53300 (cannot connect now); turn it on with compaction.backpressure.enabled = true.

Backups: object storage, encrypted, drilled

The backup job we recommend is a CronJob that triggers RedDB’s built-in pg_basebackup-equivalent:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: reddb-backup
  namespace: reddb-system
spec:
  schedule: "0 3 * * *"
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 7
  failedJobsHistoryLimit: 14
  jobTemplate:
    spec:
      backoffLimit: 2
      template:
        spec:
          restartPolicy: OnFailure
          serviceAccountName: reddb-backup
          containers:
            - name: backup
              image: registry.reddb.io/reddb-backup:1.8.4
              args:
                - --source=reddb-0.reddb-headless.reddb-system.svc:5432
                - --destination=s3://acme-reddb-backups/$(NODE)/$(DATE)/
                - --encrypt-key-arn=$(BACKUP_KMS_ARN)
                - --retention-days=35
              env:
                - name: NODE
                  value: reddb-0
                - name: DATE
                  valueFrom:
                    fieldRef:
                      fieldPath: metadata.name
                - name: BACKUP_KMS_ARN
                  valueFrom:
                    secretKeyRef:
                      name: reddb-backup-kms
                      key: arn

Three things to verify in your environment:

Backups land where you think. Run aws s3 ls s3://acme-reddb-backups/ --recursive | head the morning after the first run. A backup job that exits 0 but writes to the wrong prefix is the textbook silent failure.
Retention matches compliance. --retention-days=35 covers a 30-day compliance window with a week of buffer. If you are in a 7-year regulated industry, you want a tiered policy (daily for 30 days, monthly for 7 years) — the simplest implementation is two separate CronJobs writing to two separate prefixes with two separate lifecycle rules on the bucket.
You have done a restore drill recently. Backups that have never been restored are wishes. Once per quarter, restore to a scratch cluster, run a smoke query, write down the wall-clock time. If your RTO target is 1 hour and your last drill took 2 hours, you have a problem you would rather discover now than during the incident.

Network policy: deny-by-default, then carve out

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: reddb-data-plane
  namespace: reddb-system
spec:
  podSelector:
    matchLabels:
      app: reddb
  policyTypes: [Ingress]
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              app.kubernetes.io/name: application
      ports:
        - protocol: TCP
          port: 5432
    - from:
        - namespaceSelector:
            matchLabels:
              app.kubernetes.io/name: monitoring
      ports:
        - protocol: TCP
          port: 9180
    - from:
        - podSelector:
            matchLabels:
              job-name: reddb-backup
      ports:
        - protocol: TCP
          port: 5432
    - from:
        - podSelector:
            matchLabels:
              app: reddb
      ports:
        - protocol: TCP
          port: 5433

The deny-by-default is the important bit. Almost every incident we have helped with where someone exfiltrated data started with “the data port was reachable from somewhere we did not realize.” Carve out the three flows the cluster actually needs (app → data, monitoring → admin, backup-job → data, replica → replica) and nothing else.

Secrets: KMS-backed, not literal Kubernetes Secrets

The encryption-at-rest key is the one secret that, if leaked, ruins your night even if you rotate it immediately (because the leaked version covers everything written under it). Treat it accordingly:

Store the KMS ARN in a Kubernetes Secret, not the key material itself. RedDB unwraps at boot via the pod’s IAM role / workload identity.
Rotate the KMS key on the documented schedule (we walked through the zero-downtime mechanism in the vault rotation post).
Audit access to the KMS key, not just access to the cluster. Most Kubernetes audit dashboards do not surface “someone called kms:Decrypt against the production key from outside the cluster’s IAM role,” which is exactly the alert you want.

Things this checklist deliberately skips

A few topics that come up and that you should think about separately, not as part of “is the cluster ready”:

Multi-region. The above is for a single region. Multi-region is the topic of the disaster-recovery post — RPO/RTO trade-offs, async vs synchronous replication, and the operational tax both impose.
Connection pooling. Run a pooler (PgBouncer, RedDB’s bundled reddb-pool) between your app and the data plane. Sizing the pool is a function of your app, not the cluster.
Schema migrations. Use whatever your team already uses (Atlas, sqitch, plain psql scripts in CI). The cluster does not care, and any opinion this post had would just be cargo-culted on day one.
Capacity planning. Out of scope for “get to production-ready.” Re-evaluate every quarter after you have real traffic.

TL;DR

Pin the image, use a StatefulSet, give it fsync-honoring SSD storage at 3× hot working set, watch four metrics, run encrypted backups to object storage and actually restore them, deny-by-default networking, KMS-back the at-rest key. If you can show evidence for each of those, you are production-ready. If you cannot, the cluster will tell you which one you skipped — usually around 03:00.

The companion DR post picks up from here: what to do when the region itself goes away.