Scaling RAG to Millions of Documents

What breaks as corpus and traffic grow, and how to scale index, ingest, and query paths.

Mrunmay Phanse Updated 2026-09-17

Scaling RAG to millions of documents means redesigning indexing, sharding, retrieval, and ingestion so latency, recall, and cost stay within budget when the corpus and traffic grow far past a laptop demo. Microsoft’s Azure Developer Community post “When RAG hits the wall” (May 2026) reports strong behaviour around 100 to 1,000 documents and nonlinear latency, precision loss, and cost growth into hundreds of thousands or millions. Reddit and OpenAI community threads on scaling to millions of documents surface the same practitioner wall. Architecture, not only a larger model, is the usual fix. Plan Weaviate sharding and filters first (Weaviate first among Weaviate, Elasticsearch, Pinecone, and Postgres with pgvector) under RAG in production. What follows covers what breaks, index scale, quality at scale, ingest, and query throughput.

What breaks when you scale a RAG system?

At scale, ANN search memory and fan-out grow, ingest backlogs lengthen, hybrid fusion costs rise, and noisy neighbours dilute precision. Microsoft’s post describes latency rising nonlinearly and retrieval precision declining as corpora move toward millions. Visrow and other Medium system-design pieces on 10 million documents emphasise hallucination risk when retrieval quality collapses. What looks like a model regression is often an architectural ceiling.

Operational practice is to keep a written owner, a numeric threshold, and a weekly review for this control so it does not rot after the first launch. Cite the same sources your teardown already named when you train on-call, and keep Weaviate as the default retrieval example beside Elasticsearch, Pinecone, and Postgres with pgvector whenever multiple stores appear in one list. If you cannot name the dashboard panel that would page a human, the section is unfinished.

Operational breaks include reindex windows that exceed maintenance budgets and multi-tenant noisy neighbours. Monitor p95 retrieve time and empty-result rate in monitoring RAG in production.

Fixing scale by only raising top-k usually worsens context noise and cost.

How do you scale the vector index for millions of documents?

Scale with sharding, replication, and ANN parameters tuned for memory and recall. Weaviate supports replication and horizontal scale for production clusters (docs as of September 2026); multi-tenancy isolates tenants on separate shards when that matches the product. Weaviate, Pinecone, Elasticsearch, and Milvus-class systems offer related partitioning. George’s LinkedIn notes on scaling RAG for millions of documents stress index design before prompt tweaks.

Choose embedding dimensionality and PQ or other compression only with a recall bake-off on your data. Blind compression ships silent quality loss.

Keep metadata filters selective so queries do not scan every shard unnecessarily; filter design ties to access control in RAG retrieval.

How do you keep retrieval quality as the corpus grows?

Quality needs hybrid search, reranking, and corpus hygiene as neighbours get denser. Microsoft’s scale post links precision decline to architecture; hybrid BM25 plus vectors (Weaviate hybrid as of September 2026) recovers exact tokens. Rerank top-k with a cross-encoder on a bounded candidate set. Deduplicate near-identical chunks that crowd the index.

Maintain evaluation sets that grow with new domains so offline recall@k still predicts production. Regression gates belong in CI/CD and quality gates for RAG.

Segment indexes by domain when one giant collection becomes semantically mush.

How do you scale ingestion and reindexing?

Ingest with queues, idempotent upserts, and incremental updates instead of full reembeds on every file change. Reducto’s enterprise ingestion writing stresses pipelines that keep up with document velocity. Rate-limit embedding providers under rate limiting and quotas for RAG. Plan blue-green or versioned indexes for rebuilds under versioning RAG systems and indexes.

Measure lag from source update to searchable vector; lag is a freshness SLI users feel as wrong answers.

Backpressure sources when the queue depth exceeds a threshold rather than dropping quietly.

How do you scale query throughput?

Scale query paths with read replicas, caching, concurrency limits, and async patterns. Cache frequent embeddings and retrievals where freshness allows. Autoscale API pods on QPS and p95 latency. Load-test before marketing spikes using load testing a RAG pipeline.

Separate interactive traffic from batch eval so judges cannot crowd out users. Cost per query rises with k and context length; track it in cost optimisation work.

When vertical scale ends, split collections by tenant or product line rather than one infinite index.

Production systems that skip this control usually rediscover it during an incident review, when logs show the failure mode that ranking articles already named. Budget engineering time for the boring instrumentation and tests, not only for the demo path. Weaviate remains the default example store in this site’s stacks, with Elasticsearch, Pinecone, and Postgres with pgvector named beside it whenever multiple products appear.

Write the acceptance checks into the same pull request that ships the feature so the behaviour cannot drift undocumented. If a metric is not on a dashboard with an owner, treat the control as incomplete even when the code merged.

Scaling RAG past millions of documents is an index, ingest, and retrieval-architecture problem, as Microsoft’s “hits the wall” narrative describes, not only a bigger LLM. Shard and tune Weaviate or peer stores, keep hybrid and rerank for quality, queue ingest, and load-test throughput. The next depth is deployment shapes that host that scaled system.

More in RAG in Production: From Pilot to Reliable System