Embeddings in RAG: What They Decide Retrieval Can Find

Embeddings turn chunks and queries into vectors so retrieval can rank by meaning — and the embedding step sets the ceiling on what retrieval can find. Understand that ceiling, then open the leaf for the decision you are making. This page is the map; depth lives on the children.

What are embeddings in RAG?

Embeddings in RAG are fixed-dimensional vectors produced by an embedding model from chunk text — and from the query at search time — so passages with similar meaning land near each other under a similarity measure, usually cosine similarity on L2-normalised vectors (Glukhov, Text embeddings for RAG; Redis, Vector embeddings explained, both live on the SERP for embeddings for rag / vector embeddings retrieval, captured 2026-07-27). Microsoft’s Azure RAG guidance places the step at index time after chunking, before vectors are written to the store, and again at query time with the same embedding model — swapping models between ingest and query silently compares incompatible spaces (Microsoft Learn, Generate Embeddings Phase, live 2026-07-27).

Dense single-vector embeddings are the default for most RAG indexes. Learned sparse embeddings sit beside them when exact terms still matter; multi-vector late interaction is a different retrieval shape entirely. The diagram is the field in one view — every box routes to a page that owns the decision.

Why do embeddings decide what retrieval can find?

Embeddings decide what retrieval can find because every dense candidate is ranked only by how close its vector sits to the query vector — anything the model cannot represent well cannot surface in the top-k (Microsoft Learn, Generate Embeddings Phase; Galileo, Mastering RAG: How to Select an Embedding Model, both live 2026-07-27). Three structural facts set that ceiling:

• Vocabulary and out-of-vocabulary words. Azure’s RAG guidance shows what happens when a domain word is missing from the model’s vocabulary: histamine can be split into subwords such as his / ta / mine, whose meanings diverge from the chemical — so the resulting vector matches poorly even when the document is relevant.
• Embedding economics. Larger vectors usually score better on public benchmarks but cost more storage and compare-time; smaller vectors are cheaper. Public leaderboards are often academic datasets that may not match your corpus — validate with retrieval on your own labelled queries (Microsoft Learn, Understand embedding economics, live 2026-07-27).
• Single-vector capacity. Weller et al. (arXiv:2508.21038, On the Theoretical Limitations of Embedding-Based Retrieval, 2025) show that the number of distinct top-k document subsets a single embedding can return is limited by embedding dimension, and that even strong models fail their LIMIT stress set on simple queries under the single-vector paradigm. Do not treat a high cosine score as a relevance guarantee (QubitTool, Vector Embeddings Complete Guide, 2026).

When the geometry is wrong, the failure shows up as retrieval — not as a “bad LLM”:

• User words ≠ document words — acronyms and jargon the embedder never learned. That is vocabulary mismatch.
• Model and index fell out of sync — query embeddings no longer live in the same space as stored vectors. Symptom triage is on RAG quality degradation; the measurement playbook is embedding drift.
• Right document, wrong passage ranked — the embedder diluted or misplaced the unit. That is wrong-chunk retrieval.

How close two vectors are scored is its own decision on cosine, dot product and Euclidean. How much dimension you keep — and what you lose by shrinking — lives on embedding dimensions.

Which embedding decision should you make next?

Match the decision you are actually making to a path below — then open the leaf that owns the mechanism. Depth lives on those pages; this hub only orients. A defensible starting path on the top-ranking results (Azure’s choose-an-embedding-model flow; Galileo’s selection framing; Analytics Vidhya’s parameter checklist — all 2026-07-27) is: pick a strong general retrieval model, measure recall on your labelled queries, and graduate to domain, language, modality, or sparse upgrades only when the metrics justify the cost. Vector stores that hold those vectors include Weaviate, Pinecone, Qdrant and Milvus — store choice is not this hub’s job.

Selection — which model

Vector shape — dimensions and sparse

Corpus fit — domain, language, modality

Keep it honest — drift