How to Choose an Embedding Model for RAG

There is no single best embedding model for RAG: public leaderboards shortlist candidates, and only a labelled sample of your own retrieval queries picks the model that matches your corpus, languages, modality, dimension budget and latency. GreenNode’s 2026 roundup states that conclusion plainly: there is no single best model, only the one aligned with retrieval goals, data domain and operational limits. Milvus’s CCKM bake-off of ten 2025 to 2026 models opens the same way: the right model depends on data types, languages, document lengths and whether you need dimension compression. Weaviate’s choosing guide (Monigatti, Hasan and Hwang, 4 June 2024) turns the principle into a procedure: identify the use case, pick a skeptical MTEB Retrieval baseline, evaluate on your own labelled objects, iterate, and treat fine-tuning as an optional last squeeze. Vector stores that hold the resulting vectors include Weaviate, Pinecone, Qdrant and Milvus.

What is the best embedding model for RAG?

The best embedding model for RAG is the candidate that maximises measured retrieval quality on your labelled queries under your latency, storage and privacy constraints, not the name that currently tops a public average. GreenNode closes its five-model guide with that sentence almost verbatim. Milvus builds CCKM because MTEB alone does not answer cross-modal retrieval, hard cross-lingual search, needle finding inside long documents, or quality after Matryoshka truncation to 256 dimensions. PE Collective’s 2026 write-up of six models tested on real documents (OpenAI text-embedding-3-large, Cohere embed-v4, Voyage voyage-3-large, BGE-M3, Nomic Embed v2, Jina Embeddings v3) is useful as a shortlist, not as a universal ranking you can paste into a purchase order.

Candidate families that dominate guides for this query include hosted API embedders such as OpenAI text-embedding-3-small and text-embedding-3-large, Cohere Embed, Voyage, and Gemini Embedding; strong open retrieval models such as BAAI BGE-M3, E5, Snowflake Arctic Embed, Jina embeddings, Qwen3 Embedding and Nomic Embed; and small local baselines such as all-MiniLM-L6-v2. Baseten’s open-source picks name Qwen3 8B Embedding as an all-around option, EmbeddingGemma under one billion parameters, and Nomic Embed Code for code. Current MTEB ranks are not invented here: leaderboards move monthly, so open the Retrieval task tab on the day you decide, then validate on your set. What an embedding is, and why it caps retrieval, lives on embeddings in RAG; model-family properties live on embedding models for retrieval.

A purchase that skips your own eval usually fails in one of three ways. The model never saw your jargon and splits domain terms into subwords whose meanings diverge from the concept. The model is English-centred while your queries are not, which is why Milvus treats hard cross-lingual retrieval as a first-class CCKM axis rather than a footnote. The model fits the leaderboard and still blows the latency or storage budget at your corpus size, which is why GreenNode’s 50 to 100 millisecond query-embed rule of thumb and Milvus’s 12x storage example from 3072 to 256 dimensions both belong in the shortlist conversation. Those failure modes are why the next question is not who won MTEB last week but how a defensible selection procedure actually runs.

GreenNode’s role-of-embeddings walkthrough is worth keeping in mind while you shortlist: the model turns the query into a vector, retrieves the nearest chunks, and only then feeds context to the language model, so efficiency and domain fit show up as wrong neighbours long before they show up as a bad generated sentence. Continuous adaptation sits at the end of that chain: if your corpus vocabulary moves, yesterday’s “best” model can quietly stop matching without any deploy that looks related to embeddings.

Named product profiles for individual vendors sit under the infrastructure cluster once you have a shortlist: OpenAI embeddings, Cohere embeddings, Voyage embeddings, and open-source embedding models. Selection here stays at the decision layer so those profiles do not have to re-teach the purchase rule.

How do you choose an embedding model for RAG?

You choose an embedding model for RAG by matching the candidate to your retrieval distribution first, then scoring it on a labelled sample, instead of buying the top public name. Weaviate’s June 2024 guide states step 1 as identifying the use case: modality (text only or multimodal), subject domain (coding, law, medical, multilingual), and deployment mode. In most cases a general-purpose model for the desired modalities is the sensible baseline. Step 2 opens MTEB as a landscape overview: prefer the Retrieval task tab over a generic all-tasks average, and read model size, memory, embedding dimensions and max tokens beside the score.

The deciding question buried under every top-five list is whether this candidate matches your retrieval distribution better than it ranks on a public board. Same language or languages as the corpus, vocabulary that covers domain terms, modality that matches what you index, and a max-token limit that fits your chunk size are the distribution checks. Weaviate notes that for common RAG paragraph chunks, models with about 512 max tokens are often enough, while longer embedded units need larger windows. Galileo’s selection guide adds private versus public hosting, cost, search latency, language support and text granularity as the operational axes that sit beside raw retrieval score. Pinecone’s rundown makes the same point operationally by walking Cohere, OpenAI and a local model through one shared test-data path before anyone declares a winner.

If the distribution check fails, stop shopping on the leaderboard. Domain vocabulary the general model never learned belongs on when general embeddings fail on jargon. Cross-language questions belong on multilingual RAG. Images and text in one index belong on multimodal embeddings. Exact identifiers that dense neighbours keep missing often need hybrid BM25 and vector search or sparse embeddings rather than a larger dense model. MongoDB’s tutorial adds a further filter: even a strong retrieval score is the wrong buy if mean embedding latency is unacceptable for your query path. When the distribution check passes, the factor table decides which constraint usually dominates the remaining shortlist.

What factors matter when choosing an embedding model for RAG?

The factors that matter when choosing an embedding model for RAG are domain and vocabulary fit, language coverage, dimensions versus storage, max tokens versus chunk size, open-source versus API hosting, and latency plus dollars per embed. GreenNode’s criteria list the same axes as retrieval accuracy, latency and throughput, vector dimensionality and storage cost, and domain fit and adaptability. Galileo walks vector dimension, private versus public models, cost, search latency, language support, privacy and granularity. None of those guides invent a universal ranking; they weight constraints that change the index you actually ship.

Read the first row before you chase a one-point MTEB bump. If your problem is domain vocabulary the general model never learned, swapping to the next public name rarely fixes it. If the problem is storage at one hundred million vectors, Milvus’s 12x example from 3072 to 256 dimensions at float32 (about 1.14 TB to 95 GB in their illustration for a Milvus collection) shows why Matryoshka truncation is a first-class constraint, not a footnote. Depth for truncation lives on embedding dimensions. GreenNode also notes licensing and deployment flexibility: an open weight you cannot run at your required queries per second is not a free win over a hosted API.

Milvus’s other CCKM axes change the factor weights further. Cross-modal retrieval matters when queries are text and documents are images, or the reverse. Key-information retrieval inside 32,000-token documents matters when your chunks are long on purpose. Those axes do not replace domain fit; they decide whether a text-only English shortlist was the wrong shortlist from the start. The next practical question is how to score two candidates when both look plausible on paper.

How should you evaluate an embedding model on your own data?

You should evaluate an embedding model on your own data by building a labelled retrieval set, holding the chunker and top-k fixed, and scoring candidates with ranking metrics such as precision at k, recall, MRR or NDCG. Weaviate recommends starting with about 50 to 100 hand-labelled objects and shows, on a tiny eight-pastry toy set, that text-embedding-3-small and text-embedding-3-large can tie on average precision 0.5 and average recall 0.58 across three subjective queries while returning different rankings. That tie is the point: leaderboard position did not decide the winner, and you need more objects and more queries before the numbers stabilise.

MongoDB’s tutorial (Apoorva Joshi) builds an evaluation set from about 350 WikiHow documents broken into chunks, embeds with candidate models, and reports mean embedding latency in milliseconds beside retrieval quality. For LLM-as-judge answer checks, that tutorial asks for a test set of at least 50 samples that includes common questions, edge cases and adversarial inputs. TigerData’s open-source workflow uses Ollama to run local embedders and pgai Vectorizer to manage embeddings in PostgreSQL, with Paul Graham essays as a demo corpus, so the comparison is reproducible without a hosted API. Pinecone’s embedding-models rundown similarly walks downloading test data, generating embeddings with Cohere, OpenAI and a local model, then comparing performance on one shared path.

When you compare latency, measure both document embedding at ingest and query embedding at search time. MongoDB’s mean-latency-per-chunk number is an ingest-oriented signal; GreenNode’s 50 to 100 millisecond rule of thumb is a query-path signal. A model that is cheap to batch overnight and expensive per interactive query can still be the wrong buy for a chat assistant even if its retrieval scores look strong on a weekend bake-off.

Hold everything else constant when you swap only the embedder: same chunks, same metadata filters, same top-k the generator will see. Public-versus-private benchmark literacy sits on RAG benchmarks; the wider evaluation spine is how to evaluate a RAG system. A candidate that wins your labelled set still has to survive production drift when the corpus or the model version moves, which is why detecting embedding drift belongs in the selection plan rather than only in the incident log. Before you trust any shortlist, you also need a clear answer for what MTEB is actually good for.

Libraries such as ranx make precision at k, MRR and NDCG cheap to compute once you have query-to-relevant-document pairs. Start with dozens of well-curated pairs if that is all you can label this week; grow toward a few hundred before you trust a production bake-off. The pastry toy set is a teaching scaffold, not a production sample size. Weaviate’s own write-up says you would need to increase both the number of data objects and the number of queries before the averages become decisive.

When is the MTEB leaderboard useful for RAG?

The MTEB leaderboard is useful to shortlist retrieval-task candidates and to compare published model size, dimensions and max tokens; it is not the final purchase score. Weaviate tells you to open the Retrieval task tab, not a generic all-tasks average, and to stay skeptical of self-reported numbers because some models may have seen public benchmark data during training. Datasets on the board may not match your documents. MongoDB points at the Retrieval Text Embedding Benchmark (RTEB) as another fairness-minded shortlist that mixes open and private datasets, and still insists you validate on your data for cost and latency.

Milvus argues MTEB is largely single-language and text-centric: it does not tell you how a model behaves on cross-modal retrieval, hard cross-lingual search, long-document needle finding, or quality after dimension compression. Their CCKM suite exists to cover those four axes. MMEB adds multimodal coverage but, in Milvus’s framing, skips hard negatives, so models can score high without proving they handle subtle distinctions. Baseten’s FAQ on trusting benchmarks asks the same question from the open-source side: leaderboards are a start, not a substitute for your workload. PE Collective’s real-document test is another reminder that a public average can hide failure on the document shapes you actually index.

Task tabs matter as much as the headline score. Weaviate’s guide warns that an all-rounder average can hide a weak retrieval model, and that language-specific or domain-specific tabs (for example law) are the right refinement when your corpus is not generic English web text. Score alone without model size and memory can also mislead: retrieval often scales with size, but so does latency, and larger models can overfit public sets while underperforming in production.

Leaderboards also move monthly. As of September 2026, re-check the live Retrieval tab on the day you decide, and treat any blog’s pasted rank table as a dated snapshot rather than a permanent fact. Shortlist two or three models that fit your modality and language, then spend the evaluation budget on your labelled set. That discipline leads directly to the hosting question: whether those candidates should run as an API or as an open-weight model you operate yourself.

Should you use an open-source or API embedding model for RAG?

Use an API embedding model when text may leave your environment and you want zero model-hosting operations; use an open-source embedding model when privacy, offline operation, cost at scale, or fine-tune control dominate. Galileo’s private-versus-public framing: hosted APIs buy availability and painless upgrades, at the price of rate limits and data leaving your network. TigerData frames open-source as a way to stop paying a per-token embedding tax, regain control over the embedding process, and keep data local, while admitting that evaluation of open models is time-consuming enough that many teams default to proprietary APIs. Baseten’s open-source guide asks whether open models are as good as closed ones and how to optimise them for latency and throughput, which is the right question once privacy or cost has already forced self-hosting.

GreenNode lists both camps among its 2026 options: open families such as E5, BGE and Mistral Embed beside OpenAI text-embedding-3 and Cohere Embed. PE Collective’s six-model test mixes hosted and open weights on the same real-document workload, which is the shape of a fair bake-off. Concrete baselines used elsewhere remain useful starting points rather than winners: hosted text-embedding-3-small (1536 dimensions; OpenAI lists 0.02 US dollars per one million tokens as of the pricing page you verify the day you budget) when you want it working today; local all-MiniLM-L6-v2 (384 dimensions, zero API fee plus your CPU or GPU time) when cost or privacy wins. Building the path that embeds chunks into a store is covered on how to build a RAG pipeline; dollar modelling sits on RAG pipeline cost.

Fine-tune control is the other reason open weights win some purchases. GreenNode notes that open models such as E5 or BGE can be adapted with proprietary question-answer or document pairs, including with LoRA-style parameter-efficient updates, while many hosted APIs expose little or no embedding fine-tune surface. If your roadmap already assumes domain adaptation, that hosting fork should be decided before you fall in love with a closed model’s public retrieval score.

Vector stores that hold either class of vector include Weaviate, Pinecone, Qdrant and Milvus. Weaviate’s model-provider integrations cover many hosted and local embedders so the store does not lock the selection. GreenNode’s infrastructure note that names Weaviate, Pinecone, Chroma and Qdrant is a reminder that embedding width and indexing strategy interact: a 3072-dimension vector at float32 costs more to store and compare than a 384-dimension local baseline even before ANN parameters enter. The remaining selection forks are whether to shrink dimensions for storage and whether to fine-tune when a fair general model still fails your jargon.

When should you shrink dimensions or fine-tune embeddings for your domain?

Shrink dimensions when storage and compare cost dominate and a measured recall drop is still acceptable; fine-tune or domain-adapt when a fair general model still fails your jargon after a fair eval. Weaviate treats fine-tuning as an optional last squeeze after an off-the-shelf model is close. GreenNode recommends domain-specific fine-tuning with SentenceTransformers or parameter-efficient methods such as LoRA when general web-trained embedders stay weak on internal vocabulary. Milvus’s MRL section shows why truncation is measurable: going from 3072 to 256 dimensions is a 12x storage reduction, and on their STS-B truncation test Voyage Multimodal 3.5 and Jina Embeddings v4 lost less than 1 percent quality at 256 dimensions while other models degraded more.

Depth for truncation, Matryoshka Representation Learning and dimension mismatch failures lives on embedding dimensions, truncation and Matryoshka. Depth for recovering domain vocabulary lives on fine-tuning embeddings for your domain. Guidance echoed across selection posts still applies: quantization, Matryoshka truncation, hybrid search or a reranker often beats jumping straight to a larger embedder. See combining keyword and vector retrieval and reranking before you pay for a full re-embed into a bigger model.

If truncation looks attractive on paper, treat Milvus’s STS-B truncation method as a template rather than a transferable quality number: they used 150 sentence pairs with human similarity scores from 0 to 5, generated full-dimension embeddings, then truncated to 1024, 512 and 256 and compared Spearman correlation. Re-run an equivalent test on your own pairs before you cut a production collection to 256 dimensions because a model that held quality on STS-B can still lose recall on your acronym-heavy tickets.

Changing the embedding model, its preprocessing contract or its truncated width means re-embedding the whole corpus. Do not mix generations in one index. That migration cost and the detectors that tell you the spaces have diverged belong on when to re-embed after drift. Selection without a re-embed plan is only half a decision. Milvus also notes that truncated MRL vectors are still ordinary float vectors you can insert into a collection whose dimension matches the truncated width, which is why the dimension decision and the store schema have to move together.

Once two candidates tie on your labelled set, prefer the smaller, cheaper, or more private option and keep the larger model as a measured upgrade path. Weaviate’s iterate step is explicit about that order: make the pipeline work with a lightweight baseline, then swap only when the new model beats the baseline on the same queries. That habit prevents a 12x storage bill or a GPU reservation from entering the stack before retrieval quality has earned it.

Document the winner with the date, the labelled-set size, the metrics at the prompt’s top-k, and the rejected runners-up. Without that record, the next engineer will re-open MTEB and restart the argument from zero, and the corpus will be re-embedded twice for the same decision without a paper trail that names the metric and top-k depth that actually moved.

There is no universal best embedding model for RAG. Weaviate, GreenNode, Milvus, Galileo, MongoDB, TigerData, Pinecone, PE Collective and Baseten all converge on the same purchase rule: shortlist with public retrieval boards, then score candidates on your labelled queries under your latency, storage and privacy constraints. Weaviate’s June 2024 procedure (use case, skeptical MTEB baseline, 50 to 100 labelled objects, iterate, optional fine-tune) is the citable workflow behind that rule. Match distribution first, weight the factor table second, measure on your data third, and only then shrink dimensions or fine-tune for domain. The leaves under this cluster own those last forks so the next change matches the measurement that forced it.