Designing an AI RAG Application
The vector store gets all the attention, but a RAG application is really three storage problems wearing a trench coat.
Overview
A Retrieval-Augmented Generation (RAG) application is often described as 'just add a vector database,' but a real, working RAG system is actually three coordinated storage problems: turning documents into searchable embeddings, finding the right ones fast, and getting the full original text back to actually answer the question.
Why It Exists
An LLM can't reason over documents it was never trained on and can't fit into its context window — RAG exists to retrieve just the relevant pieces of a much larger document set and hand only those to the model. This chapter exists to walk through what that retrieval step actually requires underneath the vector database: a chunking and embedding pipeline to prepare the data, a vector store to find similar chunks via k-NN search, and a separate document store to retrieve the full text those chunks point back to — the vector alone is not the answer, only the address of it.
Real World Example
A user asks a support chatbot, 'How do I reset my API key?' The application embeds that question into a vector, searches the vector store for the k nearest matching chunk-embeddings from the indexed support documentation, and gets back a handful of chunk IDs ranked by similarity. Each ID is then used to fetch the actual chunk text from a document store, and only that retrieved text — not the whole documentation set — gets passed to the LLM along with the original question.
The Three Storage Problems Inside a RAG Application
Chunking and Embedding Pipeline
Source documents are split into manageable chunks (by size, section, or semantic boundary) and each chunk is converted into an embedding vector by a model — this pipeline runs ahead of time, not at query time.
Vector Store for Similarity Search (k-NN)
Chunk embeddings are indexed in a vector database (see Vector & Time-Series Databases and AI/RAG Vector Search at Scale) so that, given a query embedding, the k nearest matching chunks can be found quickly via approximate nearest-neighbor search.
Document Store for Retrieving Full Context
The vector store returns chunk identifiers and similarity scores, not the full original text — a separate document store (or the same database, in a smaller system) holds the actual chunk content, keyed by that identifier.
Caching Repeated Queries
Many real user questions are close variations of the same handful of common queries — caching retrieval results (and sometimes full LLM responses) for frequently-asked questions avoids repeating the embedding-and-search round trip every time.
Diagram
From a user's question to the text an LLM actually sees
User question
Embed the question
same embedding model as the pipeline
Vector store: k-NN search
returns chunk IDs + similarity scores
Document store: fetch full chunk text
by ID
LLM answers using retrieved text
Common Mistakes
Treating the vector store as if it returns exact, guaranteed-correct matches rather than approximate ones
Why: HNSW and similar vector indexes are approximate nearest-neighbor structures by design — they trade a small amount of accuracy for speed, so the top result is usually, not always, the truly closest match.
Fix: Retrieve more candidates than strictly needed (a slightly larger k) and let a lighter-weight re-ranking step, or the LLM itself, handle the final selection.
Re-computing embeddings and re-running vector search for every query, including near-duplicate or repeated questions
Why: Common questions get asked over and over in most real applications — recomputing the same embedding and search work repeatedly wastes both latency and compute cost.
Fix: Cache retrieval results for frequently seen queries, invalidating the cache when the underlying document set changes.
Deferring RAM and scale planning (from AI/RAG Vector Search at Scale) until the vector index has already grown large enough to be an emergency
Why: By the time RAM cost is visibly a problem, migrating to Product Quantization or a disk-based index like DiskANN is a much bigger, more disruptive project than if it had been planned for from the start.
Fix: Estimate expected embedding-count growth during initial design, and choose an index strategy that has headroom for that growth, not just the current dataset size.
Interview Questions
What are the two separate pieces of data a RAG application needs to retrieve, and why can't the vector store provide both?
It needs both a way to find which chunks are relevant to a question (via k-NN search over embeddings) and the actual text content of those chunks to give the LLM. Vector stores are optimized for fast similarity search over embeddings, not for storing and serving large amounts of full text efficiently, so a separate document store typically holds the actual chunk content, keyed by the identifier the vector search returns.
Why might the single most similar chunk returned by a vector search not actually be the best chunk to answer the user's question?
Vector indexes like HNSW perform approximate nearest-neighbor search, trading a small amount of accuracy for much faster search — the top result is usually close to truly optimal but not guaranteed to be. Retrieving a slightly larger set of candidates and re-ranking them (with a simpler model or heuristic, or letting the LLM weigh multiple retrieved chunks) compensates for this approximation.
How would you design the caching layer for a high-traffic RAG application to reduce both latency and LLM cost, while keeping answers accurate as the underlying documents change?
I'd cache at two levels: retrieval results (which chunks were returned for a given or similar query) and, more aggressively, full LLM responses for queries that are exact or near-duplicate matches of previously answered ones. The key challenge is invalidation — both caches need to be invalidated or bypassed when the underlying documents they depend on are updated, re-chunked, or re-embedded, otherwise the cache serves an answer based on now-stale source material. I'd tie cache entries to a version identifier for the document set (or the specific documents a cached answer drew from) so that a document update invalidates exactly the cache entries that depended on it, rather than either never invalidating or invalidating everything on every change.
Production Best Practices
Do
✓Retrieve slightly more candidates than needed and re-rank, rather than trusting the single top vector match.
✓Cache retrieval and LLM results for frequently repeated or near-duplicate queries.
✓Plan for embedding-count growth and RAM/index strategy from the initial design, not after it becomes a problem.
Don't
✗Don't treat vector search results as exact or guaranteed-correct matches.
✗Don't recompute embeddings and re-run search for every query without any caching.
✗Don't defer scale planning for the vector index until growth has already made it an emergency.
Comparison
| Purpose | Query Type | Returns | |
|---|---|---|---|
| Vector Store | Find relevant chunks | k-NN similarity search | Chunk IDs + similarity scores |
| Document Store | Retrieve full content | Lookup by ID | Full chunk text |