Load Testing a RAG Pipeline

Load testing a RAG pipeline means driving concurrent, realistic queries through the full retrieve-then-generate path and holding p95 / p99 latency, error rate and throughput — so you find which stage saturates before users do. A hello-world Locust script that posts “hi” to /chat is not enough. This page is what to measure, how to run a realistic harness, and what bottleneck the test usually reveals. It sits under RAG in production.

Why does RAG break traditional load testing assumptions?

A conventional API returns one payload and you measure round-trip time. A RAG request retrieves context from a vector store or search index, then generates tokens — often streamed. Qainsights’ performance-testing guide (captured 28 July 2026) states the consequence: a single “request duration” number hides time-to-first-token versus generation rate. MaxoPerf’s RAG pipeline load-testing docs add that under concurrency the vector database and the LLM can saturate independently — a store that handles high RPS alone may queue once concurrent RAG requests each need an ANN scan plus a long generation. Two testing surfaces matter (Qainsights): performance (speed under load) and quality (is the answer grounded?). Quality gates belong to evaluation and CI; this page is the performance half.

What metrics should a RAG load test hold?

Hold the tail and the failure rate, not only the average. Ranking pages agree on the same core set:

• p95 and p99 end-to-end latency — LoadForge and MaxoPerf emphasise tails over means; averages hide the queries that blow the budget.
• Error rate — including empty answers and missing sources when your script validates the body, not only HTTP status (LoadForge).
• Requests per second / sustainable throughput under a stated latency SLA (Chatnexus KPI framing).
• Per-stage latency when instrumented — embed, retrieve, generate — so a climb in p95 is attributable.

Chatnexus also tracks resource utilisation and cost per 1,000 queries as companion dimensions; the dollar model is at RAG cost. Qainsights’ example repo uses p95 < 6000 ms at 10 virtual users on a Gemini File Search demo as a starting point for that setup — not a universal target. Qainsights explicitly warns against copying a blog latency number; set thresholds from your own baseline. Stage budget diagnosis lives at latency.

How do you run a realistic RAG load test?

Parameterise with a real query CSV, ramp virtual users, and cover scenarios that isolate stages — not one happy-path chat. MaxoPerf treats a realistic query file as a prerequisite; LoadForge’s scenarios include retrieval-heavy search without full generation, authenticated multi-turn chat, and document ingestion versus query traffic. Tools named across the top-ranking results: Locust (Microsoft Learn’s RAG chat sample and LoadForge), k6 (Qainsights and MaxoPerf), and Azure Load Testing / JMeter (Microsoft LLMOps workshop via DeepWiki). Warm caches deliberately or disable them (MaxoPerf’s cache-effects section) so you measure the cold path you care about. Dev.to’s RAG pipeline stress tester adds concurrent users plus query-category coverage. No published QPS ceiling is universal — report what your run sustains under your p95.

Where do RAG pipelines bottleneck under load?

Under concurrency the bottleneck is usually one of four — LoadForge’s common-bottlenecks list and MaxoPerf’s stage notes agree on the map:

• Embedding throughput limits — the embed API or local encoder cannot keep up with query (and ingest) volume.
• Vector-database saturation — ANN search queues; MaxoPerf notes the vector DB often becomes the secondary bottleneck after LLM capacity is confirmed.
• LLM token-generation delays — provider quotas, long contexts, or slow models dominate end-to-end time under light load and still dominate heavy concurrent loads.
• Background ingestion contention — reindex or embed jobs share compute with live queries and steal latency.

Ailog’s 2026 performance study and Chatnexus add self-inflicted latency: oversized context and always-on rerank. Growth axes (corpus size, QPS, freshness) are at scaling; fixing the stage budget is at latency. This section is what the load test is for — revealing which of the four moved first.

How do you set SLOs and gate RAG load tests in CI?

Set p95 / p99 and error-rate thresholds from your own baseline, not a blog number — Qainsights states that warning explicitly. Published examples are setup-specific: Qainsights’ demo uses p95 < 6000 ms; MaxoPerf docs illustrate failure criteria such as p95 > 8000 ms fail and error rate > 3% fail — attribute those as their examples. Wire the load test into CI alongside the offline quality gate (Qainsights’ both-gates pattern; DeepWiki’s Microsoft LLMOps continuous benchmarking). After ship, the live watch is monitoring; the pre-merge quality half is CI/CD.