Indexing for RAG: Structures, Trade-offs and Limits

What the index does to recall, latency and memory — the layer most RAG tutorials skip entirely.

Indexing for RAG is the structure that stores document embeddings so approximate nearest-neighbour search can return candidates without scanning every vector — and the structure you pick sets the recall, latency and memory ceiling. Find the index type or the production constraint, then open its leaf.

What is indexing in RAG?

Indexing in RAG turns embedded chunks into a searchable structure — a Flat scan, an HNSW graph, or IVF partitions — so a query embedding can find near neighbours without comparing every stored vector. Meilisearch’s RAG indexing guide frames that job as organising data so the retriever locates relevant text without scanning the whole knowledge base; Machine Learning Mastery’s RAG series (Palomares, March 2025) adds that indexing strategies speed similarity search while trading some accuracy for that speed. The stages, in order:

Embeddings already exist — chunks have been vectorised. Model choice lives on embeddings; this hub assumes vectors are ready to store.
Build an ANN index — Flat keeps raw vectors; HNSW wires a multi-layer graph; IVF assigns vectors to clusters. Exact Flat search is the accuracy ceiling and the latency floor.
Optional quantization — compress vectors so more of them fit in RAM. The memory dial and its recall cost live on quantization.
Query (± filters) — traverse the structure for the query embedding; when metadata predicates are present, filtering changes which candidates are legal — see metadata filtering.
Hand candidates to retrieval — the shortlist becomes what scoring and top-k decisions work on. Scoring itself is retrieval, not indexing.

What structure the embeddings land in is the only decision that matters at this stage. Chunking and embedding set what can be found; retrieval and generation can only work with candidates the index returned.

RAG indexing stages. Embeddings enter. Index structure: Flat, HNSW, or IVF. Optional quantization. Query with optional metadata filters. Candidates handed to retrieval. — Indexing as five decisions. Embeddings arrive from upstream; Flat, HNSW or IVF decides how neighbours are found; quantization trades memory for approximation; filters constrain the legal set; candidates feed retrieval.

Indexing as five decisions. Embeddings arrive from upstream; Flat, HNSW or IVF decides how neighbours are found; quantization trades memory for approximation; filters constrain the legal set; candidates feed retrieval.

Why does the index choice change recall and latency?

The index choice changes recall and latency because every approximate nearest-neighbour structure trades how many vectors it actually compares for how much memory and build cost it spends. Flat is exact and linear in corpus size. HNSW navigates a multi-layer graph and typically keeps high recall in RAM. IVF searches only a few clusters (nprobe) and needs rebuilds when the data distribution drifts.

Milvus’s IVF-versus-HNSW comparison (as of their IVF guide) states the bands most teams use as a starting point: without compression, IVF recall can reach 95%+; HNSW usually lands around 98%+. Under heavy metadata filtering — when 90%+ of vectors are excluded — Milvus reports HNSW traversal can degrade toward a near full-graph scan because the remaining graph fragments, while IVF stays more stable by coarse-filtering at the centroid level first. That is why filtered RAG and multi-tenant predicates are not a free add-on to “just use HNSW” — see metadata filtering and multi-tenancy.

Memory and writes pull the other way. BigData Boutique’s HNSW-versus-IVFFlat guide (Syn-Hershko, 29 May 2026) puts HNSW memory overhead at roughly 2–5× the raw vectors against about 1.1× for IVFFlat, and notes HNSW absorbs inserts without a rebuild while IVFFlat recall drifts as new points land in stale clusters. On an AWS pgvector indexing deep-dive they cite — about 58k vectors at 1536 dimensions on a db.r5.large — sequential scan sat near 650 ms while HNSW landed around 1.5 ms and IVFFlat around 2.4 ms; those figures are illustrative for that workload — verify on your corpus before you rely on them. When memory is the bottleneck, product quantization compresses further: Milvus’s IVF_PQ note puts recall near 70% at about a 64:1 compression ratio, recoverable toward 90%+ with milder compression — the recipes live on quantization.

Get the index wrong and retrieval never sees the right passage — the failure shows up as a wrong answer, not as an index error. Measure with recall@k on a labelled set, and keep deletes and re-embeds on updates and deletes so removed documents stop being retrieved.

Filtered search changes the winner

When predicates exclude most of the corpus, Milvus’s IVF-versus-HNSW comparison favours IVF’s centroid-level coarse filter over HNSW graph traversal. Measure filtered recall before you lock the default — metadata filtering →

What kind of index do you need? Find the structure

Match the question you are actually asking to a leaf below — structures first if you are choosing how neighbours are found, production constraints if filters, tenants or deletes are the problem. Depth lives on those pages; this hub only orients.

Structures — how neighbours are found

These three leaves own the ANN decision: which graph or partition you build, and how hard you compress the vectors that sit inside it.

Production constraints — filters, tenants, deletes

These three leaves own what breaks after the structure is chosen: predicates that collapse the candidate set, isolation between tenants, and the update path that leaves stale vectors behind.

Don’t know where to start?

If RAM allows and documents change continuously, start with HNSW. If memory or heavy filters dominate, read IVF and Flat next — then measure filtered recall before you ship.

What is indexing in RAG?

Indexing in RAG is the step that turns embedded chunks into a searchable structure — Flat, HNSW, or IVF — so approximate nearest-neighbour search can return candidates without scanning every stored vector. The structure you pick sets the recall, latency and memory ceiling for everything retrieval and generation see afterward.

How does indexing work in RAG?

Embedded chunks are written into an ANN index (Flat scan, HNSW graph, or IVF clusters), optionally quantized to shrink memory, then queried with the question embedding — plus any metadata filters. The index returns candidates; retrieval scores and selects top-k; generation only sees that shortlist.

HNSW vs IVF — which should I use?

Use HNSW when you need high recall, continuous inserts, and enough RAM for the graph — Milvus’s comparison puts typical HNSW recall around 98%+, and BigData Boutique (May 2026) notes HNSW absorbs writes without rebuilds. Prefer IVF when memory is tight, the corpus is large and mostly static, or heavy metadata filters exclude most vectors — IVF’s cluster coarse-filter stays more stable under high filter ratios. Measure filtered recall on your own labelled set before locking either default.

When should you update a RAG index?

Update whenever documents are added, changed or deleted in a way that should change answers, and whenever you switch embedding models — old vectors will not match new ones. Meilisearch’s RAG indexing guide puts the rule as: if a human expects the new information in the answer, the index must reflect it. The delete path and tombstone behaviour are covered on the updates leaf.