Making an AI coding agent read your entire codebase at the start of every session can quickly drain your API budget and make its responses less useful.
[dvc-benchmark value="170K vs 19.5M" label="Annual token reads" detail="Article comparison from a solo-developer workflow; directional result, not universal benchmark."]
- Token consumption (annual): ~170K (retrieval) vs. ~19.5M (full-context)
- Ideal tech stack: SQLite + FTS5 (full-text search)
- Performance benchmark: 95.2–95.4% Recall@5 on LongMemEval, achieved by a hierarchical retrieval system rather than raw keyword search alone
- Cloud vector DB required: no, unless you need to scale to an enterprise setup
Why Pasting Full Context Drains Your API Budget
Large LLM context windows can make it tempting to put a whole codebase into every prompt. That may work for a one-off debugging session, but it becomes costly over a long project. You pay for the same tokens each time the agent reads them, often across dozens of interactions a day.
With so much boilerplate in the prompt, the agent can also lose track of what you actually asked.
Token Costs: 19.5M vs. 170K Annually
The 170K-versus-19.5M comparison comes from one solo developer's logs for a daily coding workflow, not a lab benchmark. Use it as a rough guide, not a prediction for your setup. The idea is simple: on startup, a local agent loads a short, high-level summary of the project's current state. It retrieves detailed specs, previous bug fixes, and past decisions only when a specific question calls for them.
A Local-First Setup: SQLite + FTS5
You don't need a costly cloud vector database to give an agent persistent memory. For a solo developer or small team, one SQLite file alongside the agent can do the job.
SQLite's built-in FTS5 extension can return full-text search results in under a millisecond. It also avoids the work of generating, storing, and updating vector embeddings for every small code change.
When Full-Text Search Beats Semantic Vector Search
Cloud vector databases are useful when you need to search for loosely described concepts. Coding agents, though, often need to find exact strings: a function name, a distinctive variable, or a known error code. Those searches call for keyword matching rather than an approximation based on semantic similarity.
FTS5 works well for that. With tens of thousands of memory entries, a local SQLite search can be faster than a remote vector store because there's no network round trip. Your project history also stays on your machine.
A minimal schema uses SQLite's external-content pattern: a regular table stores the rows, and an FTS5 virtual table indexes them by rowid rather than by a separate text key.
CREATE TABLE memory_meta (chunk_id INTEGER PRIMARY KEY, file_path TEXT, commit_hash TEXT, updated_at TEXT, content TEXT);
CREATE VIRTUAL TABLE memory_fts USING fts5(content, content='memory_meta', content_rowid='chunk_id');
A search for a specific error string takes one line: SELECT chunk_id, file_path FROM memory_fts WHERE memory_fts MATCH 'ECONNREFUSED' ORDER BY rank LIMIT 5; No embeddings or API call are needed, and a local-file query can return results in under a millisecond.
How to Chunk Markdown and Code
Putting raw, unstructured code files into a database makes retrieval less useful and can confuse the agent. Split files into chunks that follow their structure and logic.
- Markdown chunking: split documentation at H2 and H3 headings so each result stays focused on the feature it describes.
- Code chunking: keep functions, classes, and distinct logical blocks together instead of splitting at arbitrary line counts.
- Metadata tagging: attach file paths, commit hashes, and timestamps to each chunk. This helps the agent identify the version it's reading and avoid relying on outdated logic.
When SQLite Is No Longer Enough
SQLite can handle thousands of memory fragments with ease, but very large monorepos shared by dozens of developers may need more. If the agent mainly needs to trace connections between services, text search alone can miss the wider architecture.
Moving to pgvector and Graph Databases (Neo4j)
A vector or graph database starts to make sense when relationships across codebases matter more than individual code snippets. Large production setups may use pgvector to search millions of tokens for conceptually similar code blocks.
Graph databases such as Neo4j can help map dependencies, like how a change to an API gateway affects three downstream microservices. That added infrastructure may be justified for a complex enterprise system, but it's usually more than a personal coding assistant needs.
Building Your Own Memory Setup (Decision Matrix)
Start with a small setup, track token use, and add complexity when retrieval accuracy starts to suffer. A practical path looks like this:
- Phase 1, solo or prototype: use one SQLite file with FTS5. Focus on well-structured Markdown and precise code chunks. There are no cloud costs, and queries run locally.
- Phase 2, growing project: keep SQLite for exact keyword searches, then add embeddings from lightweight open-source models for queries that need semantic search.
- Phase 3, team scale: use a dedicated backend with pgvector and Neo4j to map complex relationships and support many team members reading memory at once.
My first agent memory setup went straight to embeddings because every tutorial made them seem essential. Each retrieval needed an embedding API round trip before the agent could start reasoning. But for a question like "where did I handle the ECONNREFUSED retry logic," a keyword search for the function name would have found the right chunk immediately. For that kind of query, removing the vector layer and searching FTS5 directly brought retrieval time down from a few hundred milliseconds to under a millisecond.
Choosing which coding agent to build this around matters less than choosing the memory layer. That decision shapes how your costs change as the codebase grows.