Scaling RAG to Millions of Documents

Scaling RAG means surviving three independent growth axes — corpus size (index memory and sharding), query traffic (throughput and replicas), and freshness (reindex without starving live queries). A prototype that works on about a thousand documents, Redis’s January 2026 POC framing, fails on architecture — not on a bigger box. This page is what breaks on each axis and how to scale each.

What breaks when you scale a RAG system?

Three axes break independently. Treating them as one “make it bigger” problem is why teams keep firefighting the wrong layer.

• Corpus size — chunk count drives vector memory and ANN-graph size. Exact nearest-neighbour stops being viable; approximate search trades recall for latency. Ranking pages such as Chitika’s “Scaling RAG to 20M Docs” guide and ML Journey’s scaling write-up list the symptoms as rising retrieval latency, falling accuracy, and climbing infrastructure complexity — without a single universal percentage that holds across corpora.
• Query traffic — if ingest and query share one process, embedding jobs steal latency from live answers. Alex Nechyporenko (March 2026) and Redis (January 2026) both treat offline indexing versus online retrieval as the first architectural split.
• Freshness — keeping millions of chunks current needs a reindex path with an explicit delete/cleanup story. Redis’s production write-up names the deletion failure: source docs go away while orphaned vectors keep ranking until a cleanup job runs.

The stage-by-stage latency budget lives at RAG latency; how hard to push the online path under synthetic load is at load testing. This page is which axis to scale.

How do you scale the vector index for millions of documents?

You stop treating the index as one in-process structure and design for approximate search, memory, and horizontal partition.

Memory scales with N × D. Float32 vectors alone are N × D × 4 bytes before the ANN graph. One million vectors at 1,536 dimensions is about 6.1 GB of raw floats; the same count at 384 dimensions is about 1.5 GB. Hélain Zimmermann (February 2026, updated March 2026) uses that dimension warning when choosing embedding models at ≥1M documents — it is arithmetic, not a vendor latency claim, and HNSW/IVF graph overhead sits on top of it.

ANN indexes (HNSW, IVF and similar) make million-scale search feasible. Tune recall versus latency on queries from your own logs, not on a demo set — Zimmermann’s practical sequence is vendor defaults first, then measure p50/p95 against a small ground-truth set.

Shards and replicas do different jobs. Shards partition embeddings across nodes for capacity; replicas duplicate shards for availability and read throughput. Zimmermann warns that teams over-shard early and then pay cross-shard latency; Chitika’s scaling FAQ makes the same locality point — keep related data (tenant, domain, geography) on as few shards as practical. Add replicas when QPS rises; add shards when node memory or per-shard latency is the constraint.

Distributed indexing is why vector-database choice matters at this size. Stores that teams reach for here include Weaviate, Pinecone, Qdrant and Milvus — capabilities and limits per product live under vector databases. This page does not crown one.

How do you keep retrieval quality as the corpus grows?

Pure dense retrieval misses exact identifiers and rare terms more often as the corpus grows, so production systems run sparse/BM25 alongside vectors and rerank a short merged list.

The durable pattern, carried by Redis (January 2026), Zimmermann (2026), Nechyporenko (2026) and ML Journey: dense search and sparse search in parallel, fuse the ranked lists (Reciprocal Rank Fusion is the common score- agnostic merge), then apply a cross-encoder or similar reranker on the top candidates. Redis points at published hybrid work (arXiv:2410.20381) reporting recall gains on the order of 1% to 9% versus vector search alone depending on implementation — enough to matter at scale, not a guarantee for your corpus. Measure on your queries.

How fusion and BM25 scoring work is at hybrid search; this page is only why million-document corpora force the pattern instead of “vectors only.”

How do you scale ingestion and reindexing?

Separate the offline ingest path from the online query path so embedding workers never steal latency from live traffic — the split Anyscale (Ray ingest tutorials), Nechyporenko and Redis all treat as non-optional once the corpus is large.

• Chunk, then batch-embed — queue documents, embed in batches, upsert into the vector index from workers you can scale with queue depth. Chunking strategy itself is at chunking.
• Prefer append-only / versioned index rows — Zimmermann’s pattern keeps a canonical store, writes new chunk versions, and flips an is_active flag rather than mutating vectors in place so the index stays reproducible. Broader versioning of corpus, embeddings and prompts is at versioning.
• Pick batch versus CDC by freshness SLA — Redis describes the usual path: nightly full reindex until stakeholders feel yesterday’s content missing, then change-data-capture or stream updates when staleness is a measurable business problem. Do not start on the most complex sync.
• Delete explicitly — deprecate the source and remove or deactivate the vectors. Additive sync alone leaves orphaned embeddings that keep ranking.

How do you scale query throughput?

Scale the online path on its own: more read replicas and API workers for concurrent queries — do not grow shards only because QPS rose.

Zimmermann’s guideline is replicas first for reads, shards when memory or per-shard latency demands them. Nechyporenko separates an async API layer so retrieval and generation can queue under load. At the edge, rate limits and load balancing protect shared embedding and LLM budgets — depth at rate limiting. Semantic caching cuts repeated LLM calls when queries are near-duplicates; Redis (January 2026) cites Bamsa et al. (arXiv:2411.05276) for reductions up to 68.8% of LLM calls in the workloads that paper studied — verify on your traffic, and read the failure modes at caching.

Measure p95 before you tune. Synthetic stress belongs at load testing; the per-stage budget is at latency. Honest constraint: once retrieval is healthy, generation usually dominates wall-clock time — fix the index and hybrid path first, then inference serving.