Skills as data — storing metadata, runs, and learned refinements in RedDB

The flat-file problem, take two

The previous post in this pillar made the case for putting agent memory in RedDB instead of MEMORY.md. The same argument applies to skills, and the failure modes rhyme.

A Claude Code skill lives at ~/.claude/skills/<name>/SKILL.md. Its frontmatter declares a name and a description, and the harness uses the description as a coarse-grained trigger — when the user says something that matches the description, the skill is suggested. A typical install has dozens of skills (tdd, harden, optimize, caveman, huashu-design, …) and the trigger logic is plain prompt engineering: “here are the skills, here are their descriptions, pick one.”

Three things that approach cannot do:

• Rank by past success. If optimize produces good output 9 times out of 10 on Vue work and polish only 3 of 10, the harness has no way to know.
• Compose. Two skills that frequently run in sequence (tdd → code-review-and-quality) are not linked anywhere.
• Improve over time. A skill that failed because its trigger description was too vague does not get fixed by the system that observed the failure.

Treat skills as rows in a database and all three become normal queries.

Schema

Four tables. Names match what Claude Code already exposes so the mapping is obvious.

-- One row per installed skill. Mirrors the SKILL.md frontmatter.
CREATE TABLE skill (
  id            TEXT PRIMARY KEY,             -- e.g. "tdd"
  version       TEXT NOT NULL,                -- semver of the SKILL.md
  description   TEXT NOT NULL,                -- the frontmatter description
  triggers      TEXT[] NOT NULL DEFAULT '{}', -- keywords / patterns
  embedding     VECTOR(1024) NOT NULL,        -- embedded description+triggers
  installed_at  TIMESTAMPTZ NOT NULL DEFAULT now()
);

-- One row per invocation. Append-only.
CREATE TABLE skill_run (
  id          TEXT PRIMARY KEY,               -- ulid
  skill_id    TEXT NOT NULL REFERENCES skill(id),
  session_id  TEXT NOT NULL,
  args        JSONB NOT NULL,                 -- the task the skill received
  started_at  TIMESTAMPTZ NOT NULL DEFAULT now(),
  ended_at    TIMESTAMPTZ,
  outcome     TEXT,                           -- success | failed | abandoned
  notes       TEXT                            -- model's post-run reflection
);

-- Learned refinements. Updated whenever a run produces a new lesson.
CREATE TABLE skill_lesson (
  id          TEXT PRIMARY KEY,
  skill_id    TEXT NOT NULL REFERENCES skill(id),
  pattern     TEXT NOT NULL,                  -- "when args.lang=rust, prefer X"
  weight      REAL NOT NULL DEFAULT 1.0,
  embedding   VECTOR(1024) NOT NULL,
  updated_at  TIMESTAMPTZ NOT NULL DEFAULT now()
);

-- Edges between skills that ran in sequence within one session.
CREATE TABLE skill_edge (
  from_skill  TEXT NOT NULL REFERENCES skill(id),
  to_skill    TEXT NOT NULL REFERENCES skill(id),
  count       INTEGER NOT NULL DEFAULT 1,
  PRIMARY KEY (from_skill, to_skill)
);

CREATE INDEX skill_run_skill_idx ON skill_run (skill_id, started_at DESC);
CREATE INDEX skill_embedding_idx ON skill USING hnsw (embedding vector_cosine_ops);
CREATE INDEX skill_lesson_embedding_idx ON skill_lesson USING hnsw (embedding vector_cosine_ops);

The four tables cover the three things the flat-file world cannot: skill_run for ranking by outcome, skill_edge for composition, skill_lesson for refinements.

The fit query

This is where the database earns its keep. Given a new task description, return the top-K skills ranked by a blend of semantic fit and past success rate:

WITH q AS (SELECT embed($1) AS v),
stats AS (
  SELECT
    skill_id,
    count(*) FILTER (WHERE outcome = 'success')::float
      / NULLIF(count(*), 0)                              AS success_rate,
    count(*)                                             AS runs
  FROM skill_run
  WHERE started_at > now() - interval '90 days'
  GROUP BY skill_id
)
SELECT
  s.id,
  s.description,
  1 - (s.embedding <=> q.v)                              AS fit,
  COALESCE(st.success_rate, 0.5)                         AS success_rate,
  COALESCE(st.runs, 0)                                   AS runs,
  -- blend: fit dominates when runs are low, success rate matters when we have data.
  (1 - (s.embedding <=> q.v)) * 0.6
    + COALESCE(st.success_rate, 0.5) * 0.4               AS score
FROM skill s, q
LEFT JOIN stats st ON st.skill_id = s.id
ORDER BY score DESC
LIMIT $2;

Notes worth reading:

• COALESCE(success_rate, 0.5) — a skill with no history gets a neutral prior, so a new skill is not punished for being new.
• The 60/40 blend is a knob. If the install is brand-new, push it to 80/20 toward semantic fit. After a few hundred runs, slide it to 40/60.
• interval '90 days' decays the relevance of ancient runs without explicit pruning.

Wiring it into Claude Code

The harness fires three events that are useful here: SessionStart (boot), PreToolUse/PostToolUse (every tool call, including the Skill tool), and Stop (turn end). One hook per event is enough.

1. Record installations on SessionStart

A short bash hook syncs installed skills into the skill table whenever a session boots. Drop it at ~/.claude/hooks/skills-sync.sh:

#!/usr/bin/env bash
set -euo pipefail

for dir in ~/.claude/skills/*/; do
  name="$(basename "$dir")"
  meta="$dir/SKILL.md"
  [[ -f "$meta" ]] || continue

  # Parse frontmatter (description, version) with yq.
  desc="$(yq -r '.description // ""' "$meta")"
  ver="$(yq -r '.version // "0.0.0"' "$meta")"

  curl -sS --fail --max-time 2 \
    -H "Authorization: Bearer $REDDB_TOKEN" \
    -H "Content-Type: application/json" \
    "$REDDB_URL/skill/upsert" \
    -d "$(jq -nc \
          --arg id "$name" \
          --arg ver "$ver" \
          --arg desc "$desc" \
          '{id:$id, version:$ver, description:$desc}')" >/dev/null
done

# Return empty hookSpecificOutput — this hook is a side effect.
echo '{"hookSpecificOutput":{"hookEventName":"SessionStart","additionalContext":""}}'

RedDB computes the embedding server-side at upsert time, so the hook stays one curl call per skill. Two seconds is enough budget for a few dozen skills; if you have hundreds, switch to a single bulk POST.

2. Log every run via PostToolUse

The Claude Code harness emits a PostToolUse event for every tool call. The payload includes the tool name and its arguments. Match on Skill:

{
  "hooks": {
    "PostToolUse": [
      {
        "matcher": "Skill",
        "hooks": [
          { "type": "command", "command": "~/.claude/hooks/skill-run-log.sh" }
        ]
      }
    ]
  }
}

#!/usr/bin/env bash
# ~/.claude/hooks/skill-run-log.sh
set -euo pipefail

# The harness pipes the tool-use payload as JSON on stdin.
payload="$(cat)"
skill_id="$(jq -r '.tool_input.skill' <<<"$payload")"
session_id="$(jq -r '.session_id'    <<<"$payload")"
args="$(jq    '.tool_input.args // {}' <<<"$payload")"
outcome="$(jq -r 'if .is_error then "failed" else "success" end' <<<"$payload")"

curl -sS --fail --max-time 1 \
  -H "Authorization: Bearer $REDDB_TOKEN" \
  -H "Content-Type: application/json" \
  "$REDDB_URL/skill_run/insert" \
  -d "$(jq -nc \
        --arg sid "$skill_id" \
        --arg ses "$session_id" \
        --argjson args "$args" \
        --arg outcome "$outcome" \
        '{skill_id:$sid, session_id:$ses, args:$args, outcome:$outcome}')" \
  >/dev/null || true

The trailing || true is deliberate: a logging hook must not block the agent if RedDB is briefly unreachable. The --max-time 1 budget keeps the hook invisible to the user.

3. Build skill edges in a Stop hook

When the turn ends, look at the ordered list of skill runs in this session and bump skill_edge.count for each adjacent pair. A single SQL statement on the API side handles the upsert:

INSERT INTO skill_edge (from_skill, to_skill, count)
VALUES ($1, $2, 1)
ON CONFLICT (from_skill, to_skill)
DO UPDATE SET count = skill_edge.count + 1;

After a week the skill_edge table is a usable Markov chain: given the user just ran tdd, what comes next? code-review-and-quality with weight 0.6, git-workflow-and-versioning with 0.3.

A /suggest-skill slash command

Put the fit query behind a slash command and the model can call it directly:

~/.claude/commands/suggest-skill.md:

---
description: Suggest the best-fitting skills for a task using past run data
argument-hint: <task description...>
---

Here are the top 5 skills ranked by semantic fit blended with 90-day success rate. Pick one and invoke it — do not ask the user to confirm if the top score is above 0.75.

!curl -sS --fail \
  -H "Authorization: Bearer $REDDB_TOKEN" \
  -H "Content-Type: application/json" \
  "$REDDB_URL/skill/suggest" \
  -d "$(jq -nc --arg q "$*" '{query:$q, top_k:5}')"

Usage in a session:

/suggest-skill the alignment between cards in the pricing grid feels off on mobile

The model gets a JSON list back, reads the top entry (probably polish or frontend-design), and invokes it without a round-trip. The keyword “alignment” did not have to be in any skill’s static description — the embedding does the work.

Learned refinements

Skills get better when the system that runs them remembers which patterns worked. Add a Stop hook that asks the model itself to write a one-line lesson when a run was particularly good or bad:

# Pseudo: read last skill_run for this session, if outcome=success
# and notes is non-empty, write a skill_lesson row.
curl -sS \
  -H "Authorization: Bearer $REDDB_TOKEN" \
  -H "Content-Type: application/json" \
  "$REDDB_URL/skill_lesson/upsert" \
  -d "$(jq -nc \
        --arg sid "$skill_id" \
        --arg pat "$lesson" \
        '{skill_id:$sid, pattern:$pat}')"

At suggestion time, join skill_lesson into the fit query and surface the top-1 lesson alongside the skill name:

SELECT s.id,
       s.description,
       (SELECT pattern FROM skill_lesson l
        WHERE l.skill_id = s.id
        ORDER BY l.embedding <=> q.v ASC
        LIMIT 1)                                       AS hint,
       …
FROM skill s, q
…

Now /suggest-skill returns not just which skill, but which lesson applies. A skill that learned “when args.lang=rust prefer cargo nextest over cargo test” feeds that hint back to the model before it invokes.

What you get

Question	Flat-file world	Skills-as-data
Which skill fits this task?	string match on description	vector search blended with success rate
Which skills compose well?	folklore	skill_edge Markov chain
Which skill is unreliable lately?	feel	success_rate < 0.5 AND runs > 20
Why did optimize fail last Tuesday?	not recorded	SELECT … FROM skill_run WHERE …

Total wiring: four tables, three hooks, one slash command, one API. The skills themselves do not change — they are still SKILL.md files on disk. The harness still reads them. The database sits beside the harness and turns every invocation into a row, every row into ranking signal, and every ranking signal into a better suggestion the next time around.

What is next in this pillar

• A walkthrough of building a remember-this skill end-to-end in 30 minutes — same shape, applied to conversation snippets.
• Hooks that mutate persistent state, made transactional so a half-finished skill_run insert does not leave you with a phantom row when the agent crashes mid-turn.
• An MCP server that exposes skill.* and memory.* to every MCP-compatible CLI, so Cursor and Codex can read the same tables Claude Code writes.

If you want to be told when those land, grab the Atom feed.