Hybrid Search: Combining BM25 and Vector Retrieval in RAG

Hybrid search runs BM25 keyword search and dense vector search against the same query at the same time, then fuses the two ranked lists into one, so exact tokens and semantic neighbours both reach the generator. It exists because the two methods fail in opposite ways: BM25 misses a paraphrase that shares no words with the query, and a dense retriever can bury an exact identifier, error code, or product ID under semantically similar but wrong results. Google Search itself became a hybrid system in 2015, when it added the RankBrain semantic model on top of its existing keyword index, according to Google Cloud’s own documentation. In a RAG pipeline, hybrid search sits inside retrieval in RAG: it improves the candidate list a generator reads from, most often by combining the two ranked lists with reciprocal rank fusion or a weighted blend, and it works best measured against your own labelled queries rather than assumed as a free win.

Why does vector search miss exact terms?

Dense retrieval scores semantic neighbourhood, not string identity. Product codes, error codes, SKUs, statute numbers, and rare tokens often sit in weak or crowded regions of the embedding space, so a semantically adjacent chunk outranks the line that literally contains the identifier the user typed. Google Cloud’s own explanation of hybrid search puts this in vendor terms too: semantic search can only find data the embedding model can make sense of, so arbitrary product numbers, SKUs, brand-new product names, and internal codenames fall outside what the model was trained on, a gap the industry calls “out of domain” data.

A support query for an exact error string is the textbook case: a vector index can return generic prose about browser errors, while BM25 returns the one document that contains the exact token, because it matches strings rather than meaning. Keyword-only search has the opposite blind spot: it cannot bridge a paraphrase such as “cancel my account” against a document that only says “closing your subscription,” which is why hybrid search keeps both channels rather than replacing one with the other. Redis’s engineering blog frames this as a two-way failure mode: if you only evaluate final RAG output quality, you cannot tell whether a wrong answer came from bad retrieval or bad generation, so isolating recall per retriever is the first diagnostic step.

The detection signature used across this site is a positive gap between BM25’s recall at k and dense retrieval’s recall at k on the same failing query: the lexical index can see the correct chunk, and the embedding cannot rank it highly enough. That failure mode in full detail is when the user’s words don’t match the document’s. Confirming the gap on real queries is the first step; understanding how the two ranked lists actually get merged into one is the next.

How does hybrid search combine BM25 and vectors?

Hybrid search is a pipeline of up to three stages, not a single score. The first stage runs both retrievers on the same query. The second stage fuses their two ranked lists into one. An optional third stage reranks the fused shortlist with a more expensive model.

1. Dual retrieval. The same query hits a sparse index (BM25 or another inverted index) and a dense index (approximate nearest neighbour search over embeddings) in parallel. Each side returns its own top-k candidate list.
2. Fusion. The two ranked lists become one ranking, usually reciprocal rank fusion or a normalised weighted blend, often an alpha parameter between lexical and dense.
3. Optional rerank. A cross-encoder may rescore a shortlist for precision. That stage is owned by reranking, not by fusion.

The reason fusion exists is scale incompatibility. BM25 produces unbounded positive scores shaped by term statistics, while cosine similarity typically lives in roughly the range [-1, 1]. Adding those raw numbers together lets whichever channel produces larger magnitudes dominate the other, which is the production failure that rank-based fusion is designed to avoid. A data store can keep both signals inside one collection, as sparse and dense fields on the same object, or in two separate indexes queried in parallel; the fusion architecture is the same either way, and the choice is mostly about operational convenience rather than search quality.

Some platforms build hybrid search into a single index rather than running two engines side by side. Google Cloud’s Vector Search generates sparse embeddings for the token-based half using algorithms such as TF-IDF, BM25, or SPLADE, then stores them alongside the dense embeddings in the same index, so a single query can score both. Weaviate takes a comparable approach internally: a collection carries its own inverted index for BM25 scoring next to its vector index, and a single hybrid query runs both and returns one fused ranking. Once the two lists exist, the fusion method that turns them into one ranking is the part worth understanding in detail.

How does reciprocal rank fusion work?

Reciprocal rank fusion scores a document only by where it ranked in each list; it throws the raw scores away entirely. Cormack, Clarke and Büttcher’s 2009 SIGIR paper, “Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods,” defines the sum over retrievers r as:

RRF_score(d) = sum over each retriever r of:

    1 / (k + rank_r(d))

  rank_r(d)  1-based position of document d in retriever r's list
  k          60 (the paper's own default, chosen from a pilot study)

The reciprocal of the rank drops fast near the top of a list and then flattens: the gap between rank 1 and rank 2 is large, and the gap between rank 50 and rank 51 is nearly nothing, so a document buried on one list still earns real credit once it is combined with a good rank on the other. Without the constant k, the gap between rank 1 and rank 2 would be as large as the gap between rank 1 and rank 10, so a single system’s outlier first-place vote could dominate the fused ranking even if every other system considered the document irrelevant. The paper states the purpose plainly: the constant k mitigates the impact of high rankings by outlier systems. With k at 60, rank 1 contributes about 0.0164 to a list’s score and rank 100 contributes about 0.0063, so consistent agreement across lists matters more than one loud first-place vote.

Reciprocal rank fusion ranks B above C above A, because B was decent on both lists rather than exceptional on only one. A naive sum of rank positions, by contrast, also picks B as the strongest, but that agreement is coincidental: a naive sum treats the huge gap between rank 1 and rank 50 for document A as equal in weight to the gap between rank 10 and rank 1 for document C, which is exactly the sensitivity to outliers that the 1/(k + rank) form is built to avoid. The authors picked k = 60 from a pilot study and reported that the optimum is flat across a wide range, so the constant is not something most teams need to tune. Weaviate exposes this same rank-only fusion as rankedFusion, its original hybrid algorithm from before v1.24; Qdrant ships an explicit Fusion.RRF option in its Query API; and Elasticsearch, OpenSearch, and Azure AI Search all offer an RRF retriever or ranking pipeline for hybrid queries. Reciprocal rank fusion is the default choice precisely because it needs no calibration, which raises the fair question of when a tuned, score-based alternative earns its extra complexity.

Is reciprocal rank fusion better than weighted score fusion?

Reciprocal rank fusion is the robust, zero-tuning default: no score calibration is needed, and its one constant is flat across a wide range of values, so it works reasonably well out of the box on almost any pair of retrievers. Weighted score fusion, an alpha-weighted blend of dense and sparse scores after min-max or relative normalisation, is the tunable alternative for teams that have a labelled evaluation set and want to deliberately push results toward lexical or semantic matches.

Reciprocal rank fusion is not automatically the higher-scoring option. OpenSearch’s own introduction of RRF for its Neural Search plugin measured six BEIR datasets, NFCorpus, ArguAna, FIQA, Trec-Covid, SciDocs, and Quora, comparing its existing score-normalised hybrid search against hybrid search with RRF. On average across those six datasets, hybrid with RRF scored 3.86 percent lower on NDCG at rank 10 than the score-based hybrid baseline, while median query latency improved by about 1.62 percent. That is a real trade-off, not a universal ranking-quality win: RRF traded a small amount of relevance for stability and speed on that benchmark, and OpenSearch’s own numbers make the trade explicit rather than asserting RRF is simply better.

Vendor defaults diverge on this exact question, so pinning the fusion method explicitly matters if reproducibility is a concern. Weaviate for RAG switched its own default from rankedFusion, its RRF-style method, to relativeScoreFusion in v1.24, after measuring roughly a 6 percent recall improvement over the older method on the FIQA dataset in its own internal benchmark; that score-normalised default now handles the same scale-incompatibility problem RRF solves, but by normalising raw scores to a 0 to 1 range instead of discarding them. Weaviate still exposes the older method as rankedFusion for anyone who wants it back, Qdrant exposes Fusion.RRF as an explicit option in its Query API rather than defaulting away from it, and Azure AI Search, Elasticsearch, and OpenSearch all lean on RRF-based retrievers for their hybrid pipelines. Which method wins is a property of the corpus and the query mix, which is exactly why the next question, when hybrid search is worth the added infrastructure at all, has to be answered by measurement rather than by picking a vendor’s default and trusting it.

When should you use hybrid search?

Hybrid search earns its cost when user queries mix natural language with exact tokens, such as error codes, product IDs, names, versions, or quoted phrases, or when an evaluation set shows a positive gap between BM25’s recall and dense retrieval’s recall on those specific queries. That gap has to be measured on the corpus in question, because a lift published on someone else’s dataset does not transfer automatically: OpenSearch’s own six-dataset BEIR comparison, for instance, showed hybrid search beating single-method retrieval by a wide margin per dataset, while the choice between its two fusion methods moved results by single-digit percentages in either direction depending on the dataset.

There is also a real cost side to weigh against that lift. A practitioner guide to hybrid search lists the cases where it is not worth adding: a corpus under roughly five thousand chunks where dense retrieval already hits acceptable recall on its own, queries that are exclusively conversational with no identifiers or codes to miss, no labelled evaluation set yet, since adding hybrid before measuring means there is no way to tell whether it helped, operational simplicity mattering more than the last few points of relevance, such as in an early prototype, and a stack that does not yet support a sparse index natively, where standing up a second retrieval system could dominate the engineering budget for the feature. Skipping hybrid search in any of those cases is a defensible engineering call, not a missed optimisation.

Hybrid search also cannot invent a synonym the corpus never wrote down. If a user’s query uses an acronym and every document in the corpus only ever uses the fully expanded form, with no token the two share, BM25 has nothing to match, and a dense retriever may or may not have learned the association during training. That gap belongs to query rewriting or an alias dictionary maintained alongside the index, not to more aggressive fusion tuning. Confirming that hybrid search is worth building still leaves open how it interacts with a reranking stage, which is where most production pipelines add their next layer of precision.

Does hybrid search replace reranking?

Hybrid search does not replace reranking; the two solve different problems in the same pipeline. Hybrid search raises first-stage recall by covering two retrievers’ complementary blind spots, so more of the genuinely relevant chunks make it into the candidate list at all. A cross-encoder reranker raises precision on top of that list by scoring each query-document pair directly with a more expensive model, so the most relevant chunks rise to the very top of a shorter list. The production pattern most teams converge on is retrieving broadly with hybrid search, fusing the two ranked lists, and then reranking only the top handful of results before they reach the generator.

That ordering matters because of where each stage’s cost falls. Fusion, whether reciprocal rank fusion or a weighted blend, is cheap: it operates on ranks or already-computed scores for every candidate the two retrievers returned. A cross-encoder reranker is comparatively expensive per document, since it runs a full model pass on each query-document pair, which is exactly why it is applied only to the fused shortlist rather than to the full candidate set from either retriever. Skipping the reranker is a reasonable choice only when the hybrid top-k is already clean on the metrics that matter for a given use case; where precision at the very top is worth the extra latency, depth on that second stage sits at reranking in RAG and, for the model family that does the rescoring, at cross-encoders.

How do you implement hybrid search in RAG?

Stores with native hybrid search, including Weaviate, Qdrant, Pinecone, Azure AI Search, Elasticsearch, and OpenSearch, remove most of the integration work by accepting both a keyword query and a vector query in one request and returning one fused ranking. Where a store lacks that, the alternative is running a BM25 query and a vector query client-side against separate indexes and fusing the two result lists in application code; a reciprocal rank fusion implementation is short enough to write in a handful of lines once the two ranked lists exist. Either way, both indexes have to be updated on every write, library versions have to be pinned, and recall and NDCG have to be scored on a labelled set before and after any change to the fusion method or its parameters.

A short list of mistakes accounts for most of the gap between a working hybrid search and a well-tuned one. Normalising scores before fusion when the fusion method is reciprocal rank fusion adds noise without benefit, since RRF operates on rank position and never looks at the underlying score. Setting the constant k too low, below roughly 30, over-weights the very top of each retriever’s list and reintroduces the outlier sensitivity the constant exists to prevent; the standard k = 60 is a reasonable default for most corpora and rarely needs tuning. Using the same chunk size for both retrievers wastes each one’s strength, since dense retrieval tolerates larger chunks of around 500 to 1,000 tokens while BM25 generally performs better on smaller ones, which is why some teams index the same content at two different granularities for the two retrievers. Forgetting to update both indexes on a write is the quietest failure of all: a document added to the vector index but missing from the BM25 index silently degrades hybrid recall for any query that identifier should have matched on the keyword side. And tuning an alpha-weighted linear combination on a small evaluation set produces an unreliable weight, because the optimal balance between lexical and semantic scoring is highly query-dependent; a team without hundreds of labelled queries to tune against is generally better served staying with reciprocal rank fusion’s zero-tuning default.

Hybrid search exists because BM25 and dense vector search fail on different queries for different reasons, and neither method’s blind spot is the other’s. The mechanism that makes combining them work, whether reciprocal rank fusion’s rank-only scoring or a calibrated weighted blend, is the whole reason hybrid outperforms either retriever alone on a mixed query load, and the whole reason it is worth measuring rather than assuming: what wins on someone else’s benchmark, or on someone else’s corpus, is not a guarantee for the queries a given RAG system actually has to answer. The runnable, pinned version of this pipeline is built out at RAG pipeline build tutorial.