Chunking inside the engine: when the DB owns segmentation

The standard RAG stack chunks documents in application code. A worker reads a row from Postgres, runs it through a chunker — recursive character splitter, semantic chunker, whatever this quarter’s library prefers — embeds each chunk, and writes the results to the vector store. The chunker is two hundred lines of Python that nobody on the team wrote, that everyone is afraid to touch, and that disagrees in subtle ways with whatever the next service in the pipeline expects.

This post is about what happens when that chunker moves into the database. Not as a stored procedure that imitates the Python — as a declarative rule attached to the column it segments. The rule runs inside the same transaction as the write, the chunks are queryable rows, and reranking — the other half of retrieval that usually lives in application code — becomes a query operator the planner can reason about. The comparison point is the same one we keep coming back to: an external pipeline that you have to keep in sync with the data, versus a property of the column itself.

What “chunking in app code” actually costs

If you have built a RAG pipeline, the diagram is familiar. A document arrives. A worker picks it up, applies a chunking function, embeds each chunk, writes the chunks to a vector store, optionally writes back a denormalised pointer table to the source database so you can join chunks back to documents. At query time, the application embeds the question, asks the vector store for top-k chunks, fetches the source documents from the primary store, optionally reranks the chunks with a second model, and returns the result.

Every arrow in that pipeline is a place chunking can disagree with the data:

• The document changes; the chunks have not been recomputed yet. The retriever returns chunks whose source paragraph no longer exists. We covered this drift window in a previous post — chunking is where the window is widest, because chunking is the most expensive step in the pipeline and therefore the most likely to lag.
• The chunking function changes — a new splitter, a new overlap setting, a new maximum size. Now half the corpus is chunked one way and half is chunked another way, and the retrieved top-k mixes the two with no way to tell which is which.
• The chunking function and the retrieval code disagree about how to reassemble chunks back into context. The retriever returns chunk 5 of document 17; the assembler asks for “the paragraph around chunk 5”; the chunker’s notion of “around” was never written down anywhere because it was implicit in the Python.

Each of these is a class of failure that the application is structurally bad at preventing, because the application does not own the storage and does not see the writes. The database does.

Declarative chunking, attached to the column

The mental shift is the same one we made for embeddings. Stop thinking of “the chunks for document 17” as a separate object that has to be kept in sync. Start thinking of chunking as a property of the column the chunks come from — a derived view that the engine maintains automatically, the same way it maintains an index.

The shape is a CHUNK BY clause on the column definition:

CREATE TABLE documents (
  id          UUID PRIMARY KEY,
  tenant_id   UUID NOT NULL,
  title       TEXT NOT NULL,
  body        TEXT NOT NULL CHUNK BY (
    strategy  => 'recursive',
    max_tokens => 512,
    overlap    => 64,
    separators => ARRAY[E'\n\n', E'\n', '. ', ' '],
    embed_with => 'text-embedding-3-large'
  ),
  updated_at  TIMESTAMPTZ NOT NULL DEFAULT now()
);

The CHUNK BY clause is the contract. The engine guarantees that for every committed row of documents, there exists a corresponding set of rows in a derived documents__body_chunks relation, each with the segmented text and its embedding, and that the set is consistent with the body value at the row’s current visible version. There is no worker, no queue, no outbox. The chunking and embedding happen inline with the write — or, for large documents, asynchronously with the row marked chunks_pending = true until the derived relation catches up, in which case the engine refuses to return that row from a retrieval query that requires chunks.

The derived relation is a first-class table you can query directly:

SELECT c.ordinal, c.text, c.embedding
FROM documents__body_chunks c
WHERE c.document_id = $1
ORDER BY c.ordinal;

It has its own HNSW index built and maintained by the engine. It has a foreign key to the parent document that the engine cannot let you violate. Deleting the parent deletes the chunks. Rewriting body re-runs the chunker and re-embeds, all inside the writing transaction, all visible atomically to the next reader.

The point is not that the syntax is shorter. The point is that the chunking function is part of the schema, which means it is part of EXPLAIN, it is part of migrations, it is part of the backup, and most importantly it is part of the contract the engine enforces. A row of documents whose body has been written but whose chunks have not yet been generated is a row in a defined state (chunks_pending) with defined visibility rules. It is not “a bug we will find in three weeks when a customer reports stale retrieval”.

What “consistent with the current value” actually means

The hardest part of chunking-as-derived-data is not running the chunker. It is the invariant that the chunks for row 17 always reflect the current committed body of row 17, for every transactional reader, under concurrent writes.

The engine handles this the way it handles any derived state: the chunks live under the same MVCC regime as the row they derive from. When a transaction writes a new body, it also writes the new chunk set, both tagged with the same commit LSN. Readers at an earlier snapshot see the old body and the old chunks. Readers at the new snapshot see the new body and the new chunks. There is no window — not zero-with-an-asterisk, just zero — in which a retrieval query can join the new body to old chunks or vice versa. This falls out of having a single WAL across data models: the document write, the chunk inserts, the chunk deletes, and the vector index updates all serialise through the same ordered byte stream.

For large bodies the chunking and embedding are too expensive to run inline. The engine has two modes:

• CHUNK BY (..., commit_mode => 'inline') — chunking runs inside the writing transaction. The transaction does not commit until the chunks are computed and indexed. Visibility is bound to the row’s commit. Latency is bounded by the embedder.
• CHUNK BY (..., commit_mode => 'async') — the row commits with chunks_pending = true. Chunking runs in a background lane backed by the same WAL. Until it completes, retrieval queries that require chunks treat the row as not-yet-visible (they do not return stale chunks from the previous version of the row). The default for columns wider than a configurable threshold.

async is the realistic default for production. The guarantee it gives is not “the chunks are always up to date” — that is a lie any architecture would have to tell — but rather “either you see the new row with its new chunks, or you see the old row with its old chunks, never a mix.” That is the only guarantee an application can actually program against.

Reranking as a query operator

Chunking is half of the retrieval story. The other half is reranking — the second-pass scoring that takes the top-N candidates from the first-pass retriever and re-orders them with a more expensive model. In a stitched stack, reranking is application code: fetch top-50 from the vector store, send them to the rerank API, sort by the new scores, take the top-5.

Once chunks are rows in the engine, reranking can be an operator the planner schedules:

SELECT id, document_id, text, score
FROM documents__body_chunks
WHERE tenant_id = $1
ORDER BY rerank(
  query    => $2,
  text     => text,
  model    => 'bge-reranker-v2',
  first_pass => hybrid(
    lexical => bm25(text, $2),
    vector  => 1 - (embedding <=> $3),
    weights => (0.4, 0.6)
  ),
  first_k  => 50
) DESC
LIMIT 10;

rerank() is recognised by the planner the same way hybrid() is — see the hybrid search post for the underlying machinery. The planner treats it as a two-stage top-k: the inner first_pass produces a candidate set of size first_k using the cheap hybrid score; the outer reranker is called once per candidate to produce the final score; the operator returns the top LIMIT by reranked score.

EXPLAIN on that statement makes the staging visible:

Limit  (cost=2104.50..2104.55 rows=10 width=160) (actual rows=10 loops=1)
  ->  Rerank Top-K  k=10  model=bge-reranker-v2
        first_k: 50  (capped from 100 by rerank_budget_ms=200)
        Per-candidate cost: 3.8 ms (calibrated)
        ->  Hybrid Top-K  k=50  weights=(0.40, 0.60)
              ... (see hybrid-search post)
        Reranked: 50 candidates, 10 returned
        Budget used: 190 ms of 200 ms
 Planning Time: 0.42 ms
 Execution Time: 214.6 ms

The planner pieces that matter:

• first_k is chosen by the planner, not the application. A larger first_k improves the chance the right answer is in the candidate set but spends more reranker calls. The planner picks the largest first_k that fits the per-query rerank_budget_ms, given the calibrated per-candidate cost of the chosen reranker. The application sets a latency budget; the engine spends it.
• The reranker model is named in the query and resolved like any other dependency. Models are registered in a rerankers catalog (same shape as the embedders catalog) with their endpoint, calibrated latency, and pricing. Schema migrations against the model version are the same machinery as any other schema migration.
• The first-pass scoring is reused, not recomputed. Both the lexical and vector sub-scores from the hybrid() first pass are carried on the candidate rows. The reranker sees them as features it can use (bge-reranker-v2 ignores them; some rerankers consume them as a prior). Either way, no work is repeated.

The application-side equivalent of all this is the recurring bug where the candidate-set size is hard-coded to 50, the reranker takes 6 ms per call instead of the 3 ms it took six months ago, the P95 latency on the retrieval endpoint quietly drifts past the SLO, and nobody notices until a sales engineer complains. With the operator, the budget is the configured value, and the planner adjusts first_k to fit. The number tuned for last quarter’s reranker is not in your code at all.

What changes for the application

Two things that used to be the application’s job are now the engine’s.

Chunking is no longer an out-of-band pipeline. The chunking function is a column property. Re-chunking after a strategy change is a schema migration: ALTER COLUMN body CHUNK BY (...) re-runs the chunker against every existing row, in the background, against the same WAL, with the same MVCC guarantees as any other migration. The day you change max_tokens from 512 to 768 is not the day you stand up a parallel pipeline and forklift. It is the day you write one DDL statement and watch a progress counter.

Reranking is no longer two round trips and a sort in application code. The reranker is named in the query, the budget is named in the configuration, and the candidate-set size is something the planner chooses. The half-page of TypeScript that used to call the rerank API and resort the results — gone. So is the entire class of bug where the resort used the wrong score field.

What you give up: control. If your chunker is doing something genuinely custom — semantic chunking by paragraph topic, code-aware chunking by AST boundaries, layout-aware chunking from PDF tokens — you cannot express it as strategy => 'recursive'. The escape hatch is to register a chunker as a UDF and use strategy => 'udf:my_chunker', but UDF chunkers run in a sandbox with bounded CPU and memory and cannot make network calls. For most teams the four built-in strategies (fixed, recursive, sentence, markdown) cover the corpus; for teams that need more, the UDF escape hatch exists but the bounds are real.

When you would still pick the stitched stack

There are real cases where moving chunking into the engine is the wrong call.

• Your chunker calls a remote service per chunk. A document-AI service that returns layout-aware spans, an LLM that produces summary chunks, anything that makes a network hop per chunk — these cannot run inside a database transaction without inverting your latency profile. Keep the chunker external, treat its output as authoritative, write the chunks as plain rows.
• You re-chunk constantly as part of an experiment. If you are sweeping chunking strategies as part of a retrieval-eval loop, you want chunking to be a function you call from a notebook, not a column property you migrate. The engine’s path is for the chunker you have decided on, not the one you are still searching for.
• The chunker is the product. If your differentiation is a proprietary chunking strategy you ship as a library, the strategy belongs in your code, not in your customer’s database. The engine path is for teams whose differentiation is what they do with the chunks, not how they make them.

For everything else — the long tail of “we just need recursive 512/64 chunks with cosine recall and a reranker” — the stitched stack is paying ongoing operational cost for flexibility nobody is using. The schema column gives you the same chunks with no pipeline, no drift, and a query planner that can see what you are asking for.

That is the whole pitch. Chunking is a property of the column it segments. Reranking is an operator the planner schedules against a budget. The Python glue layer that used to live between Postgres and a vector store does not get smaller — it does not exist.