You ship: Agentic retrieval over a 52-document corpus, compared with top-5 retrieval on 10 questions. Builds on:Day 11, Day 12.
What you'll build today
After Day 12, WorkBot's agents read the knowledge base through 2 tools that match the words of a question against 8 policies. A real customer's knowledge goes well beyond that, with manuals, IT guides, frequently asked questions and HR guides, and the answer to one question can be spread over several documents.
The usual approach is retrieval-augmented generation (RAG), which finds the 5 passages closest to the question and pastes them into one prompt. Today you see where that one-shot retrieval fails, and you build agentic retrieval instead. You give the model 4 tools that work like a filesystem, so it can search, list, read and grep the knowledge base until it has the answer. Along the way you learn what an embedding is and how a vector search finds text by its meaning.
By the end of the day WorkBot has a 52-document knowledge base and one set of tools shared by the chat and the agents. You also have a comparison of 10 questions answered both ways, with what each approach cost.
By the end of the day you can
Understand why brute-force retrieval-augmented generation (RAG) fails, and when to use agentic retrieval
Implement the virtual filesystem pattern, so an agent can explore your data
Set up embeddings and vector search as supporting infrastructure
Topics
Why Brute-Force RAG Fails
Traditional RAG: query → embed → top-k retrieval (the k nearest chunks) → paste them into the prompt → generate
Failure modes: the wrong chunks are retrieved, the answer spans several sections, exact syntax does not rank well, and nothing explores
The fundamental limit: one retrieval cannot find its way through a large, complex corpus
When traditional RAG is fine: simple questions over a small, well-structured corpus
Agentic RAG
Give the agent tools to explore the corpus step by step
The agent can search broadly → read specific sections → refine the search → cross-reference → synthesize
This mirrors how people research: you don't read the top 5 Google results and stop
Trade-off: more model calls mean higher cost and latency, but much higher accuracy on complex queries
Concept: present your data as a filesystem the agent can browse with familiar commands
ls <path>: list the documents or sections in a directory
cat <file>: read a specific document or section
grep <pattern>: search all documents for a pattern
find <query>: semantic search for documents that match a query
Implementation: these are not real filesystem calls. They translate to database and vector store queries.
Coarse-to-fine: find (semantic search) identifies the relevant areas → ls browses their structure → cat reads specific content → grep finds exact matches
Embeddings Fundamentals
What embeddings are: text mapped to high-dimensional vectors that capture its meaning
Embedding models: Voyage AI (recommended for Claude), OpenAI text-embedding-3-small
Vector similarity: cosine similarity to rank results
Vector stores: pgvector (a Postgres extension) for production, ChromaDB for prototyping
Indexing: chunk the documents, generate embeddings, and store them with metadata
The role of embeddings in agentic RAG: they power the find tool (semantic search)
When to Use What
Traditional RAG: a small corpus, simple questions, tight latency or a tight budget
Agentic RAG: a large corpus, complex questions, accuracy that matters most, exploration needed
Long context (put everything in the prompt): a very small corpus (under 100k tokens) and simple questions
Hybrid: start with traditional RAG, and escalate to agentic retrieval when it fails
Build
Features shipped today
WorkBot: Agentic retrieval over a customer's knowledge base, policies and manuals
Build an agentic retrieval system:
Prepare a document corpus: 50 or more documents, such as a technical docs set, a knowledge base or a company wiki
Build the index: chunk the documents, generate embeddings (Voyage AI), and store them in pgvector with metadata (source, section, title)
Build filesystem tools:
find(query): semantic search that returns the 10 best matches, with their document paths and relevance scores
ls(path): list the sections and subsections of a document
cat(path): read the full content of a specific section
grep(pattern, path?): regex search across all documents or one document
Build the agent: given a question, the agent explores the corpus with the filesystem tools and finds the answer
Compare: run 10 test queries through both traditional RAG (top-5 retrieval) and agentic RAG, and compare the quality of the answers
Walkthrough
Part 1
Write the knowledge base
Agentic retrieval needs a corpus worth exploring, so you start by growing the 8 policies into a knowledge base of 52 documents in 5 sections.
Study first
Learn why WorkBot chunks by heading. Each ## section becomes one chunk, stored with its document's title and section, so a search hit is a place a person could open.
Build
Write 52 documents as content/knowledge/<section>/<slug>.md, in 5 sections: policies, manuals, it, faq and people. Move the 8 policies from Day 4 into policies, and give each document front matter (title, owner, updated) and ## headings.
Add a CLAUDE.md in the folder that names the sections and the front matter. It also says that each heading must stand on its own, because headings are what search returns.
Turn src/lib/policies.ts into a thin view of the policies section, so the Day 4 pages keep working.
Done when
The folder holds 52 documents in 5 sections, and the Day 4 policy pages still list and render the 8 policies.
Part 2
Index and search the corpus
A model cannot read 52 documents for every question, so WorkBot splits them into chunks. It indexes each chunk for 2 kinds of search, by its words and by its meaning.
Study first
Learn how embeddings and similarity work, starting from Voyage AI's page on embeddings. An embedding is a vector that places a text near texts with a similar meaning, and the cosine distance between two vectors says how far apart their meanings are. Top-k retrieval embeds the question and takes the k nearest chunks.
See where top-k fails. The answer may be spread over 2 documents, or an exact figure may sit under a heading the embedding does not reach. A section may also only make sense next to its neighbours. The agent instead searches, reads sections, and searches again from what it read.
Read the Postgres chapter on full text search, which WorkBot uses when there are no embeddings, and the pgvector extension, which stores vectors in Postgres.
Build
Write src/lib/knowledge.ts with listDocuments, getDocument(path), chunksOf(document) and indexKnowledge(embedder?). indexKnowledge writes every chunk to a chunks table with a tsvector, Postgres's indexed form of a text for word search. When it is given an embedder, it also stores a vector(1024) embedding.
Write src/lib/embeddings.ts, which provides the embedder, Voyage AI's voyage-3.5 model, only when VOYAGE_API_KEY is set. The index builds on the first find that sees an empty chunks table, and pnpm knowledge:index rebuilds it.
Write find(query, embedder, sections?). It uses cosine search when the chunks carry embeddings. Otherwise it runs a tsquery, a Postgres word query, that matches any of the question's words and ranks by how many match. Both modes search only the allowed sections.
Write grep(pattern, path?) to search the raw text.
Done when
pnpm knowledge:index reports 52 documents and their chunk count.
pnpm test covers the chunking, both search modes and the section filter, and a find with an embedder over an index built without embeddings.
Technical notes
pgvector takes one create extension if not exists vector in the schema list. Locally, run brew install pgvector for a Homebrew Postgres, or install the pgvector package for your Postgres elsewhere. The embedded PGlite database from Day 5 loads it from @electric-sql/pglite-pgvector, so the tests run the same SQL.
Without a Voyage key, search uses words only, and find says which mode it used.
The embedding's dimension is part of the schema. vector(1024) is voyage-3.5's default size, so a model with another size needs a column of that size.
The index is idempotent per document path, which means a rebuild after an edit replaces that document's chunks and leaves the rest.
Part 3
Tools scoped by section
With search in place, you give the model tools to explore the knowledge base the way a person browses files. Each set of tools is scoped to some sections, so the public web chat sees only what a customer may see.
Study first
Learn the coarse-to-fine pattern. The model uses find to locate, ls to see a document's shape, cat to read one section and grep for the exact phrase. Each tool returns only what was asked for.
Learn why the tools are scoped. A tool list built for a set of sections lets a reader see nothing outside them. The web chat is public until Day 30 adds sign-in, so it gets a scoped list.
Build
Move the tool() helper and inputSchema out of tools.ts into src/lib/tool.ts. Make run parse its input, so a schema's defaults apply to every caller.
In src/lib/knowledge-tools.ts, write knowledgeTools(sections?), which returns 4 tools. find returns the 10 best chunks with their paths and scores, and ls lists the sections at the root, a section's documents, or a document's headings. cat reads one section by path#heading or a whole document, and grep searches for a pattern, optionally in one document. Export customerSections = ["faq", "policies"].
Build 2 tool sets. employeeTools, for the terminal and the agents, holds the knowledge tools over everything, plus the calculator, page reading, notes and web search. customerTools, for the web chat, holds the knowledge tools over the customer sections, the calculator and page reading, with no notes and no web search.
Switch the investigation agent from Day 11 and the explainer's researcher from Day 12 to the knowledge tools, in place of the 2 policy tools. Their prompts say find, then cat.
Test each tool against the indexed corpus, including a path outside the allowed sections, which must come back as not found.
Done when
In the web chat, a customer question about refunds shows find then cat calls, and an answer with the policy's limit.
In the web chat, ls at the root lists only faq and policies, while in the terminal the same ls lists all 5 sections.
In the web chat on the customer tools, the find tool locates the refunds policy, cat reads the section, and the reply quotes the limit.Technical notes
Each tool's description says when to call it, with find first and then cat on the paths that look right. The order comes from the descriptions and the system prompt, not from code.
A scoped cat of a path outside its sections throws "There is no document at", the same error as a wrong path, so the tool does not reveal what exists.
grep returns matching lines with their document path and heading. Its output is capped, so a broad pattern cannot flood the context.
Part 4
Compare top-5 with the agent
You now have both ways of answering a question over the same corpus, so you can measure what agentic retrieval buys and what it costs.
Study first
Decide how to grade the comparison. An answer is correct when it states the figure or rule the document gives and cites the section that holds it, and the tokens it spends are its price.
Build
Write cli/compare-retrieval.ts with 10 questions over the corpus. For each question, get one answer from the 5 best chunks pasted into a single call and another from the investigation agent exploring with the tools. Print both with their token counts.
Mark each of the 20 answers correct or not by the rule above, and keep the list.
Done when
pnpm compare:retrieval prints 10 pairs, and your list names each question that one approach got wrong.
One question from the comparison is answered from the same corpus twice. The first answer comes from the 5 best chunks in one call, and the second from the agent exploring with find, cat and grep.Technical notes
Both runs use claude-haiku-5-5, so the comparison measures retrieval, not model size.
Expect a trade, and let your list confirm it or not. The agent spends 10 to 20 times the tokens. In return, it gets right the questions whose answer spans documents or needs a figure the 5 chunks did not include. Top-5 is enough when the answer sits in one section that the question's words name. A hybrid starts with top-5 and escalates to the agent when the first answer cannot cite a section.
