Agents · 2026-06-01 · By RedDB team · 5 min read

Context window economics — when memory beats reloading docs

Every CLI agent burns tokens re-reading the same files turn after turn. Put a number on the waste, work out the breakeven against a semantic-memory layer in RedDB, and show the SQL plus hook that flips the math for projects past a few thousand lines.

The hidden bill on every CLI agent session

Open Claude Code in a medium repo. Ask a question. Watch it Read the same five files it read yesterday. Ask a follow-up. Watch it Grep the same constants. The model is smart, the tool calls are cheap individually, and the per-turn waste hides inside a green checkmark — so nobody adds it up. This post does the addition, then shows the fix.

The fix is a semantic-memory layer that lives in the same RedDB table the rest of Pillar D uses: read once, embed, query top-k chunks per turn instead of re-reading the whole tree. The interesting part isn’t the code — it’s where the breakeven sits. For most real projects it’s around turn three.

The naive baseline: re-read everything

Call this the “stateless agent” baseline. Per turn the agent reloads:

  • CLAUDE.md and any @import-chained memory files — call it M tokens
  • a handful of source files for context — call it S tokens
  • recent diffs, git log, command output — D tokens

Per turn, input tokens: C = M + S + D. Across N turns: N · C. Cost: N · C · p_in, where p_in is the input-token price.

For Claude Opus 4.7 (Jan 2026 list) that’s roughly $15 / 1M input, $75 / 1M output. Prompt cache hits land near $1.50 / 1M. Cache is the reason this isn’t worse — but cache misses on every new turn that touches a slightly different file set, and the cache TTL is five minutes. After lunch you pay full price again.

A worked number for a 50k-line monorepo, ten-turn session, ~30k input tokens per turn (10k from rules + 15k from re-read files + 5k from logs):

input tokens   = 10 turns × 30_000 = 300_000
cost (cold)    = 300_000 / 1_000_000 × $15  = $4.50
cost (warm)    = 300_000 / 1_000_000 × $1.50 = $0.45

Forty-five cents per warm session sounds fine. Multiply by 12 engineers × 8 sessions/day × 21 working days and you’re at $907/month, and that’s the cached number. The cold-cache reality across machine restarts and overnight idle is closer to $2k/month per twelve-engineer team — for the privilege of re-reading files that haven’t changed.

The memory-backed alternative

Swap the per-turn re-read for a per-turn semantic query.

  1. Index once. A background indexer chunks every file in the repo (200–400 tokens per chunk), embeds it with voyage-3 (~$0.06 / 1M tokens, one-time amortised), writes the rows into the same memory table.
  2. Query per turn. On SessionStart (or whichever hook owns context priming), embed the conversation tail, run a <=> cosine search, pull the top-k chunks. k = 6 and chunk size 300 tokens means ~1.8k tokens of injected context vs the 15k of “re-read the whole file” — an order of magnitude shave on the source-file portion of C.
  3. Re-index on change. A PostToolUse hook on Write|Edit invalidates and re-embeds the touched files. Tiny per-edit cost; rounds to zero against any session.

The per-turn input now looks like C' = M + Q + R, where Q is the embedding query (a few hundred tokens of conversation tail) and R is the retrieved chunks. In our worked example, C' ≈ 10_000 + 800 + 1_800 = 12_600, down from 30k. The model still does the work — it just doesn’t re-read what it already knew.

The breakeven, plotted as algebra

Per session, the memory-backed path pays:

cost_memory = N · C' · p_in                # per-turn query + retrieval
            + E_query · N · p_embed        # embedding the query each turn
            + E_index · F · p_embed / R_amort   # amortised indexing

E_index is total tokens indexed, F is the change-frequency multiplier (how often files re-embed), R_amort is how many sessions the index serves before a full rebuild. For most repos R_amort ≫ 100, and p_embed is ~250× cheaper than p_in, so the indexing column vanishes.

The naive path pays:

cost_naive = N · C · p_in

Breakeven is the turn count where the constant-cost ramp-up of the memory path is paid off by the smaller per-turn slope:

N_breakeven  =  E_index · p_embed / R_amort
                ────────────────────────────────
                (C - C') · p_in  -  E_query · p_embed

Plug the worked numbers (C - C' = 17_400 input tokens saved per turn, E_query = 800, p_in = $15/Mtok, p_embed = $0.06/Mtok, E_index = 5_000_000, R_amort = 200):

numerator    = 5_000_000 × 0.06 / 200 / 1_000_000  ≈ $0.0015
denominator  = (17_400 × 15 - 800 × 0.06) / 1_000_000 ≈ $0.000261
N_breakeven  ≈ 5.7 turns

Below six turns per session, full re-read wins. Above six, memory wins, and the gap widens linearly with N. Any AFK / Ralph-style loop blows past breakeven in the first hour.

The actual code

The schema is the one from the memory-backend post, no changes:

-- already exists from D1
CREATE TABLE memory (
  id         text PRIMARY KEY,
  scope      text NOT NULL,           -- 'repo:<slug>:file:<path>'
  body       text NOT NULL,
  embedding  vector(1024),
  meta       jsonb NOT NULL DEFAULT '{}'::jsonb,
  created_at timestamptz NOT NULL DEFAULT now()
);

CREATE INDEX memory_embedding_hnsw
  ON memory USING hnsw (embedding vector_cosine_ops);

A one-shot indexer (Node, ~40 lines, runs from the repo root):

// scripts/index-repo.ts
import { readFile } from 'node:fs/promises'
import { glob } from 'glob'
import { ulid } from 'ulid'
import postgres from 'postgres'
import { VoyageAIClient } from 'voyageai'

const sql = postgres(process.env.DATABASE_URL!)
const voyage = new VoyageAIClient({ apiKey: process.env.VOYAGE_API_KEY! })
const REPO = process.env.REPO_SLUG ?? 'rdb-lair'
const CHUNK = 300                          // ~tokens, sliced by characters at 4x

const files = await glob('**/*.{ts,tsx,md,sql}', { ignore: ['**/node_modules/**', '**/dist/**'] })

for (const path of files) {
  const text = await readFile(path, 'utf8')
  const chunks: string[] = []
  for (let i = 0; i < text.length; i += CHUNK * 4) chunks.push(text.slice(i, i + CHUNK * 4))
  if (chunks.length === 0) continue

  const embeddings = await voyage.embed({ input: chunks, model: 'voyage-3' })
  const rows = chunks.map((body, i) => ({
    id: ulid(),
    scope: `repo:${REPO}:file:${path}`,
    body,
    embedding: JSON.stringify(embeddings.data[i].embedding),
    meta: { path, chunk: i },
  }))

  await sql`INSERT INTO memory ${sql(rows, 'id', 'scope', 'body', 'embedding', 'meta')}`
  console.log(`indexed ${path} (${chunks.length} chunks)`)
}

await sql.end()

The SessionStart hook that replaces the per-turn re-read:

#!/usr/bin/env bash
# ~/.claude/hooks/session-start-context.sh — emits top-k chunks for the current task
set -euo pipefail

TAIL="$(jq -r '.conversation_tail // ""' <<<"$CLAUDE_HOOK_INPUT")"
[ -z "$TAIL" ] && exit 0

QVEC=$(curl -s https://api.voyageai.com/v1/embeddings \
  -H "Authorization: Bearer $VOYAGE_API_KEY" \
  -H 'content-type: application/json' \
  -d "$(jq -n --arg q "$TAIL" '{input:[$q], model:"voyage-3"}')" \
  | jq -c '.data[0].embedding')

psql "$DATABASE_URL" -At -F $'\t' <<SQL | jq -Rs '{context: .}'
SELECT meta->>'path' AS path, body
FROM memory
WHERE scope LIKE 'repo:${REPO_SLUG}:file:%'
ORDER BY embedding <=> '${QVEC}'::vector
LIMIT 6;
SQL

Wire it up in .claude/settings.json:

{
  "hooks": {
    "SessionStart": [
      { "command": "~/.claude/hooks/session-start-context.sh", "priority": 20 }
    ],
    "PostToolUse": [
      { "matcher": "Write|Edit", "command": "~/.claude/hooks/reindex-touched-file.sh" }
    ]
  }
}

The reindex-touched-file.sh hook re-embeds just the file that changed — same shape as the indexer above, scoped to one path. Skipped here for length; it’s ~20 lines.

What to measure once it’s live

Don’t trust the model. Measure. The agent observability post drops a PostToolUse hook that captures every tool call with token counts and cost. Compare two weeks before and after:

SELECT
  date_trunc('day', started_at) AS day,
  sum(input_tokens)             AS input_tokens,
  sum(cost_usd)                 AS cost_usd
FROM tool_call
WHERE session_id IN (SELECT id FROM session WHERE repo = 'rdb-lair')
GROUP BY 1 ORDER BY 1;

Two failure modes to watch:

  • Retrieval too narrow. k = 6 chunks misses cross-cutting context. Symptom: the model asks Read for files anyway. Bump k to 12, or add a second query embedded from the user’s last message instead of the rolling tail.
  • Stale index. PostToolUse reindex misses files changed outside Claude (git pull, another editor). Fix: nightly cron that rescans git diff HEAD~24h and re-embeds.

When this is the wrong call

Three cases where the naive re-read still wins:

  • Tiny repos. Under ~2k lines, C - C' is small enough that N_breakeven jumps past 20 turns. Skip the indexer; let the prompt cache handle it.
  • Single-shot scripts. A one-question session never amortises the per-query embedding cost. Memory layer helps zero.
  • High-churn refactors. When 40% of files change per session, reindex cost climbs and R_amort collapses. In that mode, fall back to plain Read for the touched tree and only query memory for the rest.

For everything else — long-running AFK loops, multi-day feature work, anything that crosses a context-compression boundary — the breakeven is six turns. Pay it once.

Where this fits in the pillar

#claude-code#agents#context-engineering#cost#memory

© 2026 RedDB.io. AGPL-3.0 self-host · Managed Cloud.