Probabilistic

Code

Write it. Read it. Same engine.

HLL ADD distinct_users 'user:42';

HLL COUNT distinct_users;

Build it

End-to-end walkthrough.

Four steps from empty database to a real probabilistic workload — each step shows the exact code and explains what the engine is doing.

1. Step 1

   1. HyperLogLog — count distinct in 16 KB

   A HyperLogLog at precision 14 holds ~16 KB and counts distinct elements with ~0.81% standard error — independent of dataset size. Merging is associative + commutative, so you can shard counters per region and combine without coordination.

   CREATE HLL distinct_users PRECISION 14;
   
   -- Stream of events
   HLL ADD distinct_users 'user:42';
   HLL ADD distinct_users 'user:99';
   HLL ADD distinct_users 'user:42'; -- duplicate, no extra cost
   
   HLL COUNT distinct_users;
   -- → 2  (≈ 0.81% error)
   
   -- Combine sketches across regions for a global estimate.
   HLL MERGE distinct_users_global FROM
     [distinct_users_us, distinct_users_eu, distinct_users_apac];

2. Step 2

   2. Count-Min Sketch — top-K frequency

   Count-Min Sketch trades exactness for fixed memory — width × depth × 8 bytes regardless of how many keys flow through. Used for top-K queries, hot-key detection, abuse rate-limiting at scale.

   CREATE SKETCH event_frequency
     WIDTH 2048 DEPTH 5;
   
   SKETCH ADD event_frequency 'login.success' 1;
   SKETCH ADD event_frequency 'login.failed' 1;
   SKETCH ADD event_frequency 'login.success' 5;
   
   SKETCH COUNT event_frequency 'login.success';
   -- → ≈ 6  (overestimates by < 1/width with prob 1 - (1/2)^depth)

3. Step 3

   3. Cuckoo Filter — membership with delete

   Cuckoo Filter answers "have I seen this id?" with a tiny false-positive rate and supports deletes (unlike Bloom filters). Use it as a cheap pre-check before a database lookup — most negative answers don't even hit storage.

   CREATE FILTER seen_payment_ids
     CAPACITY 1000000 FALSE_POSITIVE_RATE 0.001;
   
   FILTER ADD seen_payment_ids 'pi_12345';
   FILTER CHECK seen_payment_ids 'pi_12345';
   -- → true
   
   FILTER CHECK seen_payment_ids 'pi_99999';
   -- → false (provably absent — skip DB hit)
   
   FILTER DELETE seen_payment_ids 'pi_12345';
   -- Cuckoo supports delete; Bloom filters do not.

4. Step 4

   4. Pipe a time-series into a sketch

   HLL_BUILD and HLL_CARDINALITY are aggregate-shaped UDFs. Plug into materialised views over time-series data — your weekly unique-visitor dashboard reads a tiny pre-computed sketch table, not the raw event stream.

   -- Continuous aggregate: hourly unique-visitor cardinality.
   CREATE MATERIALIZED VIEW mv_unique_visitors_hourly AS
     SELECT
       time_bucket('1 hour', timestamp) AS bucket,
       HLL_BUILD(tags->>'user_id')      AS sketch
     FROM page_views
     GROUP BY bucket;
   
   -- Query the view — sketches are materialised, counts are O(1).
   SELECT bucket, HLL_CARDINALITY(sketch) AS unique_users
   FROM mv_unique_visitors_hourly
   WHERE bucket > now() - INTERVAL '7 days'
   ORDER BY bucket DESC;

Showcases

What probabilistic can do once you bring the rest of the engine in.

Cross-model correlation, algorithm flex, multi-model patterns — things that take a stack of services elsewhere.

-- Per-region writer (1 line per page-view).
HLL ADD dau_us 'user:${user_id}';

-- Aggregator (runs at end of day, or on demand).
HLL MERGE dau_global FROM [dau_us, dau_eu, dau_apac];

-- Dashboard query.
SELECT
  bucket,
  HLL_CARDINALITY(sketch) AS dau
FROM mv_dau_daily
WHERE bucket > now() - INTERVAL '90 days'
ORDER BY bucket;

app.post('/webhooks/stripe', async (req) => {
  const evt = JSON.parse(req.rawBody)

  // Cheap pre-check.
  const seen = await db.query(
    "FILTER CHECK seen_stripe_events $1",
    [evt.id],
  )
  if (seen.rows[0].present) {
    return new Response('replay', { status: 200 })
  }

  // Real persist + record.
  await db.query("INSERT INTO webhook_events (id, payload) VALUES ($1, $2)",
    [evt.id, evt])
  await db.query("FILTER ADD seen_stripe_events $1", [evt.id])
  await processStripeEvent(evt)
  return new Response('ok')
})

CREATE SKETCH ip_5min
  WIDTH 4096 DEPTH 5
  TTL 5m;

-- Per-request hook.
SKETCH ADD ip_5min '${request.ip}' 1;

-- Alert query (runs every 30s).
SELECT ip, count
FROM (
  SELECT
    distinct_ip                     AS ip,
    SKETCH_COUNT(ip_5min, distinct_ip) AS count
  FROM (SELECT DISTINCT tags->>'ip' AS distinct_ip FROM access_log
        WHERE timestamp > now() - INTERVAL '5 minutes') s
) t
WHERE count > 1000
ORDER BY count DESC;

Recipes

Common patterns, real driver code.

CREATE HLL dau_24h PRECISION 14 TTL 24h;
HLL ADD dau_24h ${user_id};
SELECT HLL_COUNT(dau_24h);  -- ~24h cardinality in 16 KB.

CREATE SKETCH ip_freq WIDTH 4096 DEPTH 5 TTL 5m;
SKETCH ADD ip_freq ${ip} 1;
SKETCH COUNT ip_freq ${ip}; -- O(depth), ~5 hash probes.

const present = await db.query(
  "FILTER CHECK seen_ids $1", [id]
)
if (!present.rows[0].present) {
  // safe to skip the DB hit — provably absent.
  return null
}
return await db.query("SELECT * FROM rows WHERE id = $1", [id])

vs

How RedDB probabilistic compare.

Pairs with

Probabilistic earns more next to these.

The data models probabilistic compose with — a click away.

• metrics

  Time-series →

  Sketches over time-series streams: hourly DAU into HLL, top-K hot endpoints into Count-Min — bounded memory regardless of dataset size.

• fast state

  Key-Value →

  Cuckoo Filter as a pre-flight before KV reads — most negative answers stay in memory, never hit storage.

• relational

  Tables →

  A Cuckoo Filter on the existence-set saves a table scan for the 99% of ids that are not present.

Explore

Every other data model on RedDB.

One engine, 9 first-class data models. Mix as needed.

• relational

  Tables →

  Postgres-wire SQL with indexes, joins, transactions.

• json

  Documents →

  Schemaless JSON when structure is partial — query by path.

• edges

  Graphs →

  Native edges and traversals — multi-hop without a side-car.

• semantic

  Vectors →

  Auto-embed on insert, similarity search, no vector DB to sync.

• fast state

  Key-Value →

  Configuration, feature flags, hot keys — millisecond reads.

• tiered TTL

  Cache →

  Tiered blob cache built into the engine: L1 memory + L2 durable.

• metrics

  Time-series →

  Retention, downsampling, native window queries.

• workflows

  Queues →

  FIFO, priority, consumer groups — durable workflow primitives.