sqlrecall

sqlrecall

Agent memory and code recall in one local SQLite file. No services, no cloud.BM25 + dense embeddings fused with Reciprocal Rank Fusion, an optional cross-encoder reranker, and an MCP server so coding agents(Claude Code, Codex, anything MCP) can recall your notes, decisions, docs, andcode instead of guessing.

Pairs with session-memory tools likeai-memory: they remember whathappened, sqlrecall finds what was written down.

Formerly shelfmark. Renamed in 1.2.0. pip install sqlrecall replacesshelfmark-rag; the old shelfmark-* commands still work as aliases, and anexisting ~/.shelfmark data dir is picked up automatically when~/.sqlrecall does not exist.

Why this exists · Quickstart ·MCP server ·What gets indexed · Evaluation ·How it compares · Configuration

Why this exists

Agent harnesses accumulate knowledge: memory notes, ADRs, plans, standards,handoffs, and the code itself. The useful answer to most questions is alreadywritten down somewhere. sqlrecall indexes all of it into one local SQLite fileand answers "what did we decide about X / where did we handle Y" in a singlequery, with path:line citations.

sqlrecall is the retrieval engine that came out of a year of running AI codingagents daily — most recently distilled fromforgekit, an AI dev toolkit forcoding agents. Every design choice below was forced by a real failure mode inthat daily use, not picked off a paper.

Design choices that fell out of a year of measured iteration (seeeval/holdout-policy.md and inline rationale comments):

• Hybrid by default. BM25 catches identifiers and exact terms; embeddingscatch paraphrase. RRF fusion beats either alone on mixed corpora.
• Code-aware tokenization. camelCase/snake_case sub-tokens on both corpusand query sides ("create player" matches createPlayer). Measured +2.8ppcode / +3.2pp overall.
• Symbol-definition boost. Chunks whose defined symbol matches a queryidentifier get a rank-0 signal (+2.8pp code, zero regressions).
• Selective reranking. Cross-encoder rerank helps code and standards,hurts memory recall (measured -10.5pp). The rerank policy is scope-aware;memory is never reranked.
• Contextual chunk prefixes. Every chunk embeds with atype | repo | file | symbol header, not raw text.
• Freshness without wall-clock. Optional recency prior for memory scope,deterministic (reference = max mtime among candidates), so evals stayreproducible.

Quickstart

pip install sqlrecall   # or: pipx install sqlrecall

sqlrecall-build      # first run writes a starter ~/.sqlrecall/sources.yaml — edit it, then rerun
sqlrecall-query "how do we handle retry timeouts"

First build downloads intfloat/multilingual-e5-small (~120MB). Everythingafter that runs offline.

git clone https://github.com/LucasSantana-Dev/sqlrecall && cd sqlrecall
python3 -m venv venv && venv/bin/pip install -e .

venv/bin/sqlrecall-build
venv/bin/sqlrecall-query "how do we handle retry timeouts"

MCP server (agent integration)

// e.g. Claude Code: .mcp.json or ~/.claude.json
{
  "mcpServers": {
    "sqlrecall": {
      "command": "sqlrecall-mcp"
    }
  }
}

(Local checkout instead of pipx install? Use "command": "/path/to/sqlrecall/venv/bin/sqlrecall-mcp".)

Two tools:

• rag_query — full-corpus hybrid search (code, docs, commits, notes).Auto-scopes to the repo your agent is working in.
• search_knowledge — cross-project search over durable knowledge only(memory/standards/plans/handoffs/adrs; configurable viaRAG_KNOWLEDGE_SCOPE). Never reranked, by measurement.

examples/claude-code/ has the full loop: auto-recall on every prompt(UserPromptSubmit), incremental reindex on file writes (PostToolUse), driftreindex + weekly report at session start, and a nightly rebuild with an evalregression gate.

Session capture and handoffs

One command, called from each agent's lifecycle hooks, records what a sessiondid and passes a baton to the next session in the same project, acrossharnesses (start in Claude Code, continue in Codex or OpenCode):

sqlrecall-hook --agent claude-code --event session-start   # reads the hook JSON on stdin

• Capture. Prompts, tool calls (name, target, pass/fail, short output) andthe last reply of each turn go to a SQLite store next to the index(index.memory.sqlite, or index.client-<slug>.memory.sqlite under aclient's roots). Secret-shaped strings are redacted and bodies capped at16 KiB before anything is written. A cwd under a purged client is nevercaptured. Zero LLM calls.
• Automatic handoff. Every stop refreshes one open handoff per session:last request, last reply, files edited, recent commands with failures.
• Delivery. At session-start the newest open handoff from another sessionin the project (worktrees count as their main checkout) is claimed once andprinted, fenced as untrusted data; Claude Code and Codex add that output tothe new session's context. Written handoffs win over automatic ones;claiming an automatic one expires the older automatic ones. Open handoffsexpire after SQLRECALL_HANDOFF_TTL_DAYS (14).
• MCP tools. handoff_begin (leave a written handoff), handoff_list,handoff_accept. The layer always comes from the server process, neverfrom a tool argument.

Adapters: examples/claude-code/capture-hooks.json (merge intosettings.json), examples/codex/hooks.json, andexamples/opencode/sqlrecall.ts (plugin, not yet run against a liveOpenCode). SQLRECALL_CAPTURE=off disables capture;SQLRECALL_HANDOFF_ON_START=off keeps capture but skips delivery.sqlrecall-purge <client> drops the client's memory store with its index.

What gets indexed

sources.yaml declares everything (see sources.yaml.example):

Incremental reindex (indexer.py --incremental <files>) keeps writes cheap;session_chunker.py can additionally index agent session transcripts.

Evaluation

The eval harness is the part most RAG setups skip. eval/run.py scoresHit@1/3/5 + MRR per scope against a JSONL dataset; eval/check.sh gates anychange at >5pp regression vs a frozen baseline; eval/holdout-policy.mddocuments the train/holdout discipline (the holdout set is never used fortuning — numbers quoted from it are honest).

This repo ships a public, reproducible dataset (eval/dataset-public.jsonl)whose queries target this repository's own code and docs:

venv/bin/python indexer.py                 # index this repo (sources.yaml.example works as-is)
venv/bin/python eval/run.py --dataset eval/dataset-public.jsonl --label mine

Benchmark results and methodology: BENCHMARK.md.

To evaluate on YOUR corpus, write ~50 {"query", "expect_path_contains", "expect_scope"} lines, freeze a fifth of them as holdout, and wireeval/check.sh into your nightly rebuild. That regression gate is what keepsretrieval quality from silently rotting as the corpus grows.

How it compares

No cross-tool benchmark exists yet (see Methodology & honestlimitations — we'd genuinelylike to see one run). Qualitative trade-offs, no fabricated numbers:

Pick a hosted vector DB when you need multi-tenant scale across millions ofdocuments — that's a different problem than agent recall over your own repos.

Want to measure and improve retrieval ranking quality on your own pipeline,independent of any specific agent? hitgateprovides label-free regression testing for hybrid retrievers. sqlrecall andhitgate share a common hybrid-retrieval foundation (BM25 + embeddings + RRF)but serve complementary use cases: sqlrecall for zero-setup agent memory withMCP integration, hitgate for ranking evaluation and quality gates on anyretriever you already have.

Configuration

All optional — see .env.example for the full list. Highlights:

Client layers

If you work for more than one client, keep each client's business knowledgeout of the index every session reads. Declare clients in sources.yaml:

clients:
  acme:
    roots: [~/dev/acme-app, ~/notes/acme]

• Files under a client's roots, or with client: acme in their frontmatter(top level or under metadata:), are indexed into that client's own file(index.client-acme.sqlite), never the general index. client: none keeps anote general even inside a root (logged at index time). Once a client keyis present, anything else (unknown slug, list, empty, broken YAML) skips thefile instead of sending it to general.
• A query reads the general index plus the active client's file only. Theactive client comes from the process (RAG_CLIENT, or the process cwd undera client root), never from a tool-call argument, so an agent cannot switchitself into another client's layer.
• Session transcripts are routed by the cwd they recorded. Known limit: asession started outside every client root that then reads a client's filesis indexed as general. Start client sessions inside the client's root, orleave session_chunker.py off.
• BM25 is scored per file, so one client's vocabulary never shapes another'sterm statistics; results are then ranked together. With no clients declared,ranking is unchanged.
• A client's data lives only in its file and its index.client-<slug>.backup-*snapshots. Removing a client from sources.yaml while its file still existsstops the next build (its untagged files would otherwise land in general):archive or remove the file and its sources globs first. The last build'sclients are recorded in $RAG_HOME/clients.json, so this also holds for aclient with a custom db: path.
• Upgrading from a version without client layers: run one full rebuild.Chunks now store the resolved path, and rows written before keep the oldspelling until rebuilt.
• Root matching ignores case on macOS/Windows (str.casefold, close to butnot exactly the file system's own folding).

To offboard a client, purge before removing it from sources.yaml:

sqlrecall-purge acme --lexicon acme-terms.txt                       # dry run: what would go
sqlrecall-purge acme --apply --archive age1... --lexicon acme-terms.txt

The purge streams the client's index, its backups and its query-log rows intoone age-encrypted archive (never plaintext ondisk). Then it deletes them and scrubs any residue of the client from thegeneral index and its backups with secure_delete + VACUUM, so nothingsurvives in free pages or the WAL. It then checks that the lexicon's canaryterms appear nowhere in the raw bytes of what remains. Notes under theclient's roots tagged client: none (lessons you decided are general) arekept. --no-archive deletes without an archive. A tombstone keeps laterbuilds from routing the client's files into general if its globs stay.Source files are never touched.

Contributing

Issues and PRs welcome — especially a real cross-tool benchmark (see How itcompares), support for more embedding models, ornon-Claude MCP client examples. If sqlrecall saves your agent a guess, a starhelps others find it.

License

MIT