Engineering · 2026-05-17 · By RedDB team · 8 min read

Hybrid search done right: lexical, vector, and filter in one plan

A walk through the RedDB query planner fusing BM25, vector similarity, and a structured filter into a single execution plan — with EXPLAIN output, the cost model behind it, and what happens on the edge cases that trip up naive two-stage rerankers.

If you have ever wired up “hybrid search” by hand, you know the shape: send the same query to a full-text index, send it again to a vector index, post-process the two result sets in application code, then filter the merged list against your structured predicates. It works. It is also three round trips, two scoring functions that disagree about what “relevant” means, and a filter step that runs after the top-k has already been chosen — which is the wrong order if your filter is even slightly selective.

This post is about how the RedDB query planner avoids that shape. Lexical, vector, and filter are not three pipelines glued together at the application layer; they are three access paths the optimizer can mix inside one plan. The interesting part is not that the syntax is shorter. The interesting part is what the planner does when one of the three legs is degenerate — when the lexical match is empty, when the vector recall is low, when the filter is selective enough to dominate the cost model. That is where the naive stitched-together version silently returns the wrong rows.

The query, as written

The user-facing API is one statement. A tenant-scoped retrieval for a customer support agent, matching on the question text both lexically and semantically, restricted to documents the asking user is allowed to see and to a recency window:

SELECT id, document_id, body, score
FROM chunks
WHERE tenant_id = $1
  AND acl_subjects && $2          -- array overlap with the caller's groups
  AND created_at > now() - interval '180 days'
  AND match(body, $3)             -- BM25 lexical predicate
ORDER BY
  hybrid(
    lexical => bm25(body, $3),
    vector  => 1 - (embedding <=> $4),
    weights => (0.4, 0.6)
  ) DESC
LIMIT 20;

$3 is the query string; $4 is its embedding. hybrid() is not a UDF that runs after the rows have been collected — it is a planner-recognised scoring construct, the same way ORDER BY ... LIMIT k over a vector distance is recognised as a top-k operator rather than a sort. The distinction matters; we will come back to it.

What the planner actually builds

Run EXPLAIN (analyze, costs, format text) on the statement above against a 24M-chunk corpus and you get something close to this:

Limit  (cost=1842.10..1842.15 rows=20 width=128) (actual rows=20 loops=1)
  ->  Hybrid Top-K  k=20  weights=(0.40, 0.60)
        Score: 0.40 * bm25(body, $3) + 0.60 * (1 - cosine(embedding, $4))
        ->  Filtered Candidate Stream  (predicate-pushdown=true)
              Filter: tenant_id = $1
                      AND acl_subjects && $2
                      AND created_at > '2025-11-18 00:00:00'
              ->  Bitmap-AND
                    ->  Lexical Index Scan  on chunks_body_fts
                          Recheck Cond: match(body, $3)
                          Heap Blocks: exact=4112
                          Rows: 38,221
                    ->  Vector ANN Scan  on chunks_embedding_hnsw
                          ef_search: 96 (auto-tuned for k=20)
                          Probe Rows: 240
                          Rows: 218
              Bitmap-AND output: 412 candidate rows
        Rescored: 412 rows  (lexical + vector, with stored sub-scores)
        Buffers: shared hit=1894
 Planning Time: 0.31 ms
 Execution Time: 11.4 ms

Three things are worth pointing at.

First, the filter is pushed down to candidate generation, not applied after the top-k. The lexical scan and the ANN scan each emit row identifiers; the structured filter is evaluated as those identifiers stream out of the indexes, before the bitmap is intersected. If tenant_id = $1 cuts the corpus from 24M to 80k chunks, the ANN scan only walks the part of the HNSW graph that points into that tenant’s slice — the index stores a per-row tenant id in the leaf payload so the graph traversal can skip out-of-tenant nodes without a separate lookup.

Second, the two candidate streams are intersected, not unioned. The naive recipe — “take top-100 lexical, take top-100 vector, merge by some weighted score” — unions and reranks. That is the wrong default for a system where both signals are meant to point at the same row. A union biases towards documents that are strong on one signal and absent on the other, which is exactly the kind of result a human reviewing the output calls “irrelevant but on-topic”. The planner intersects by default and falls back to a union only when the intersection is too small to fill the requested k (see the edge cases below).

Third, scoring happens once, on the surviving candidates. bm25(body, $3) is computed during the lexical index scan and the sub-score is carried on the candidate row; cosine(embedding, $4) is computed during the ANN scan and similarly carried. The Hybrid Top-K operator combines the two stored sub-scores with the requested weights and runs a bounded heap of size k. There is no recompute, no second pass over the corpus, and no application-layer merge.

The cost model is what makes this stable under changing data shape. The planner estimates the selectivity of each leg — filter, lexical, vector — and chooses both the join order of the bitmaps and the ef_search parameter for the ANN scan to satisfy a target recall. It is the same machinery that picks an index in a normal OLTP planner, with two additions: an approximate-recall estimator for the ANN side, and a Zipfian estimator for term frequency on the lexical side.

What changes when a leg is degenerate

The interesting failure modes of hybrid search are not “everything works and the score is slightly off”. They are the cases where one of the three legs collapses. The planner has a defined behaviour for each.

Empty lexical match

The user asks something the corpus has no lexical overlap with. Common when the query is a paraphrase, an acronym not present in the documents, or a different language than the indexed text. The lexical leaf returns zero rows.

A naive merge would now return whatever the vector leg found, ranked purely by cosine — which is fine, except that the weighted score 0.4 * bm25 + 0.6 * cosine collapses to 0.6 * cosine for every row, and you have silently changed your ranking function in the worst possible way: the API still claims to be hybrid.

The planner handles this explicitly. When the lexical bitmap is empty and the vector bitmap is non-empty, the Hybrid Top-K operator switches to the vector-only score and emits a runtime annotation on the result set:

Hybrid Top-K  k=20  weights=(0.40, 0.60)
  Degraded: lexical_empty -> vector_only
  Rescored: 218 rows  (vector sub-score only; lexical sub-score = 0)

The application sees a degraded field on the response envelope ({ degraded: "lexical_empty" }). It is not an error — the rows are still useful — but the caller now has the signal it needs to, for example, prompt the agent for a clarifying question, or to suppress the citation footnotes that pretend the lexical signal participated.

Low-recall vector match

Pathological the other direction: the ANN scan returns very few candidates, either because the embedding is far from any cluster the index has visited at the current ef_search, or because the filter eliminated most of the ANN’s neighbourhood. The planner notices the candidate count is below a configurable floor (hybrid.vector_recall_floor, default 0.5 * k) and re-runs the ANN scan with ef_search * 2, up to a configurable ceiling. This is cheap because the HNSW state from the first pass is cached; the second pass continues the traversal from where the first stopped, rather than starting from the entry point.

If after the ceiling the vector candidate count is still below the floor, the operator widens the candidate set by unioning the lexical bitmap rather than intersecting — and again annotates the result:

Hybrid Top-K  k=20  weights=(0.40, 0.60)
  Degraded: low_vector_recall -> union(lexical, vector)
  ef_search escalation: 96 -> 192 -> 384 (ceiling)
  Rescored: 184 rows

Note that the score formula is unchanged — both sub-scores are still combined with the original weights — but the candidate set is now a union, which is the right default when the planner knows the intersection is artificially small.

Filter-dominant queries

The third edge case is the one that quietly destroys naive stacks. The filter is highly selective: a tenant with 200 chunks, or a date window that excludes 99% of the corpus. A bolt-on hybrid stack first asks the lexical index for top-100 and the vector index for top-100, then filters down — and ends up with a handful of rows, or zero, despite the underlying tenant slice containing perfectly good matches.

The planner detects this from the filter’s estimated selectivity. When the filter cardinality is below a threshold relative to k (default: filter_rows < 50 * k), the plan inverts: a filter-first execution that materialises the filtered slice and then runs lexical + vector scoring across it as a sequential pass, rather than going through the indexes. The EXPLAIN output for that variant looks different:

Hybrid Top-K  k=20  weights=(0.40, 0.60)
  Strategy: filter_first  (filter_rows=147 < 50*k=1000)
  ->  Filtered Materialize  rows=147
        Filter: tenant_id = $1 AND acl_subjects && $2 AND created_at > ...
        ->  Index Scan on chunks_tenant_btree
  Rescored: 147 rows  (lexical + vector, computed on-the-fly)

The lexical and vector scores are computed at scoring time rather than read from the index. That is more CPU per row, but the absolute row count is small, so the trade is correct. It is also the only variant that is guaranteed to find the right answer when the filter is selective — the index-first variants can miss rows entirely when neither the top-100 lexical nor the top-100 vector intersect the filtered slice.

The cost model, briefly

Three estimates drive the strategy choice:

  1. Filter selectivity. Standard stats — histograms, MCVs, distinct-count — same as any planner. Tenant id and time windows are usually well-estimated; ACL overlaps are estimated as a function of the average subject-set size.
  2. Lexical cardinality. Estimated from per-term document frequencies. A query of three rare terms might return 800 rows; a query of three stopwords-after-filtering might return zero, which short-circuits the lexical leg.
  3. Vector recall at a given ef_search. Estimated from a small offline calibration that the indexer runs once per index build and refreshes on compaction. The estimator maps ef_search to expected recall at k for the corpus’s distance distribution; the planner inverts it to pick the smallest ef_search that meets the target recall.

The strategy chosen is the one with the lowest expected cost subject to a hard recall floor. The recall floor is the protection against the “looks fast, returns garbage” failure mode: you can configure the floor down, but you cannot configure it away.

What this buys the application

Two things that used to be the application’s job are now the planner’s.

The application no longer owns the merge. There is one ranked list returned by one statement, with one consistent score, with a degraded annotation when the plan had to fall back. The half-page of TypeScript that used to weight two result sets, deduplicate, and reapply the filter — gone. So is the entire class of bug where the merge code drifted out of agreement with the index configuration.

The application no longer owns the recall/latency trade-off per call site. ef_search, the choice between intersect and union, the choice between index-first and filter-first — these are all decisions the planner makes per query, based on the statistics of the actual data and the actual query. The configuration you ship is a recall floor and a latency budget, not a set of magic numbers tuned for last quarter’s corpus.

If you are running a hybrid stack today and have ever had to explain why your top result on a slightly rephrased question is suddenly missing — or why a narrow tenant returns zero hits despite obviously having matching documents — one of the three edge cases above is the explanation. The fix is not a better merge function in application code. The fix is to push the merge into the same plan as the scans, so that the planner can see when a leg has degenerated and react accordingly.

That is the planner work behind hybrid(). The syntax is the boring part.

#engineering#rag#search#query-planner

© 2026 RedDB.io. AGPL-3.0 self-host · Managed Cloud.