AI/RAG Vector Search at Scale
A vector index that fits comfortably in RAM at 1M embeddings can become unaffordable at 100M — long before it becomes technically impossible.
Overview
Vector databases and their HNSW indexes (covered earlier in Server-Side Databases) are typically built to live entirely in RAM for fast search — a design that works well at millions of embeddings, but starts costing serious money once a RAG application's index grows past 100M+ embeddings.
Why It Exists
HNSW's graph-based index structure needs random access across the whole graph to search efficiently, which is why it's normally kept in RAM rather than on disk. RAM pressure at scale exists because embeddings themselves are just dense floating-point vectors — a 100M-embedding index at typical dimensions can require hundreds of gigabytes of RAM, and RAM is one of the most expensive resources in any cloud instance, by a wide margin over disk.
Real World Example
A RAG-powered support search tool starts with 2 million document chunk embeddings, comfortably fitting in a modest RAM footprint. As the company ingests years of historical documents, that grows past 150 million embeddings, and the fully in-RAM HNSW index alone would require an instance costing many times more than the rest of the application's infrastructure combined — pushing the team to look at RAM-reduction techniques instead of simply provisioning bigger machines indefinitely.
Example Data
Why RAM cost scales faster than embedding count feels like it should
| Embeddings | Typical Dimension | Approx. Raw Vector Size | In-RAM HNSW Feasibility |
|---|---|---|---|
| 2 million | 768 | ~6 GB | Easy, fits on a modest instance |
| 150 million | 768 | ~460 GB | Requires a very large, expensive instance |
Fitting Vector Search Into a Real RAM Budget
Why HNSW Wants Everything in RAM
HNSW's multi-layer graph traversal jumps between arbitrary nodes during search — on disk, that pattern means constant random-access reads, which are dramatically slower than RAM, so the whole index is normally kept in memory.
Product Quantization (PQ) — Compressing the Vectors Themselves
PQ splits each vector into sub-vectors and replaces each with the closest of a small set of pre-computed representative values, storing compact codes instead of full floating-point numbers — trading a small amount of search accuracy for a large reduction in memory per vector.
DiskANN — Keeping the Index on Disk, With a Smarter Layout
DiskANN restructures the graph and search algorithm specifically to make disk-based random access efficient (via SSD-friendly layout and caching), enabling billion-scale vector search without needing the entire index in RAM.
The Trade-off Is Always Accuracy or Latency for Memory
Every RAM-reduction technique gives something back — PQ trades some recall accuracy, DiskANN trades some query latency versus a fully in-RAM index — the right choice depends on how much of either the application can actually tolerate.
Diagram
As embedding count grows, RAM cost forces a trade-off, not just bigger machines
Small index (millions)
Fully in-RAM HNSW — fine
Large index (100M+)
PQ (compress vectors) or DiskANN (move to disk)
Common Mistakes
Scaling a growing vector index purely by provisioning more RAM
Why: RAM cost grows linearly (or worse) with embedding count, and can become the single largest infrastructure cost in the entire application well before other components even notice the growth.
Fix: Evaluate Product Quantization or a disk-based index like DiskANN once RAM cost becomes a significant fraction of total infrastructure spend, rather than continuing to scale vertically.
Applying Product Quantization without measuring its actual impact on search recall for the specific dataset
Why: PQ's accuracy loss depends heavily on the data's actual distribution — the compression ratio that's safe for one dataset may meaningfully degrade recall for another.
Fix: Benchmark recall against the uncompressed index on a representative query set before committing to a specific PQ compression ratio in production.
Assuming DiskANN or PQ makes vector search 'free' at any scale, with no further planning needed
Why: Both techniques reduce cost, not eliminate it — DiskANN still needs enough RAM for caching and metadata, and PQ still costs some accuracy that compounds with the other quality trade-offs the RAG pipeline is already making.
Fix: Treat these as levers that shift the cost/accuracy/latency trade-off, not as a way to avoid engaging with that trade-off entirely.
Interview Questions
Why does a vector index's RAM cost become a real concern as the number of embeddings grows?
HNSW indexes are normally kept fully in RAM for fast search, and vectors are made of many floating-point numbers each — at hundreds of millions of embeddings, the raw memory required can run into hundreds of gigabytes, making RAM one of the most expensive parts of the whole system.
What's the basic trade-off Product Quantization makes to reduce RAM usage?
PQ replaces each vector's full floating-point representation with a compact code pointing to pre-computed representative values, dramatically shrinking the memory footprint per vector, at the cost of some loss in search accuracy (recall) compared to searching the uncompressed vectors directly.
A RAG application's vector index has grown to 200M embeddings and RAM cost now dominates the infrastructure budget. How would you decide between Product Quantization and a DiskANN-style disk-based index?
It comes down to which trade-off the application can tolerate better: PQ keeps the index in RAM but compresses vectors, so it's a good fit if query latency is critical and some recall loss is acceptable and can be measured/bounded via benchmarking. DiskANN keeps full-precision vectors but moves the index to disk with a search-friendly layout, so it's a better fit if recall must stay high and a modest latency increase (still much faster than a naive disk-based search) is acceptable. In practice, I'd benchmark both against the application's actual query set and its specific recall/latency requirements rather than choosing on theoretical grounds alone, and consider whether the two techniques can be combined for very large scale.
Production Best Practices
Do
✓Evaluate PQ or DiskANN once RAM cost becomes a significant fraction of infrastructure spend.
✓Benchmark recall against the uncompressed index before committing to a PQ compression ratio.
✓Treat RAM-reduction techniques as cost/accuracy/latency levers, not free scaling.
Don't
✗Don't scale a growing vector index purely by provisioning ever-larger RAM.
✗Don't apply Product Quantization without measuring its real impact on your specific dataset.
✗Don't assume disk-based or compressed indexes eliminate the cost/accuracy trade-off entirely.
Comparison
| Storage Location | Accuracy Trade-off | Best For | |
|---|---|---|---|
| Full HNSW (in-RAM) | RAM | None — exact within HNSW's own approximation | Smaller indexes, latency-critical |
| Product Quantization | RAM (compressed) | Some recall loss | Large indexes, latency still critical |
| DiskANN | Disk (SSD) | Minimal — full precision kept | Very large indexes, some latency tolerance |