Embedding Images and Text Together for RAG

Multimodal embeddings put images and text into one vector space so a text query can retrieve figures, screenshots or diagrams, and the hard part is evaluation because text-only metrics do not tell you whether the right image neighbour returned. Weaviate’s multimodal embeddings guide walks shared-space retrieval for RAG. IBM’s multimodal RAG overview and Meilisearch’s simple guide frame the same job for production stacks. Hugging Face and Elastic show build patterns; LangChain’s multi-vector retriever targets tables, text and images together. ArXiv 2410.21943 studies multimodal inputs for industrial RAG. Vector stores that hold multimodal vectors include Weaviate, Pinecone, Qdrant and Milvus.

How do multimodal embeddings put images and text in one space?

Multimodal embeddings put images and text in one space by training encoders so that an image and a caption that describe the same content land near each other under cosine similarity. Weaviate’s multimodal guide describes that shared space as the foundation for retrieving images with text queries and text with image queries. Contrastive training on image-text pairs is the usual mechanism behind CLIP-style models that practitioners plug into RAG.

At index time you embed each modality with the matching tower of the multimodal model and store the vectors in one collection or in linked collections. At query time you embed the user text (or image) with the same model family. Neighbours can then cross modalities without an intermediate caption step, which is the native path IBM and Meilisearch contrast with bridge architectures.

Native multimodal retrieval is not the same as storing image bytes as attachments while only embedding OCR text. If only text is embedded, you have text RAG with files on the side. True multimodal embeddings make the pixels part of the retrieval geometry.

Weaviate can store and search those vectors beside text properties so filters and BM25 still apply to captions or filenames when exact tokens matter.

What is the modality gap in multimodal embeddings?

The modality gap is the systematic separation between image embeddings and text embeddings in a supposedly shared space, so nearest neighbours can cluster by modality instead of by meaning. Practitioners see text queries retrieving other text even when a better image exists, or image queries staying inside the image cloud.

The gap matters for RAG because a user question is usually text while the evidence may be a chart. If the space is gapped, you under-retrieve figures. Meilisearch and IBM both warn that multimodal systems need evaluation beyond text-only recall.

Mitigations include better multimodal training, query-side prompts that steer toward visual content, dual retrieval over caption text and image vectors, and human review of image hits. Measure modality-aware recall: fraction of questions whose gold evidence is an image and whose top-k contains that image.

Do not confuse the modality gap with ordinary wrong-chunk retrieval; the fix paths differ. Wrong text chunks belong on wrong-chunk retrieval; missing visual evidence needs multimodal geometry or a bridge.

Should you embed natively or convert images to text first?

Embed natively when visual layout carries meaning that OCR or captioning destroys; convert images to text first when a caption or OCR string already contains the answer and you want to stay on a text embedder. LangChain’s multi-vector patterns and Elastic’s multimodal RAG builds often bridge by summarising images into text for a text index, which is simpler to evaluate and cheaper to serve.

Native multimodal models cost more at ingest and need image-capable evaluation. Bridge architectures fail when the captioner invents numbers or drops axis labels, which is a common industrial failure mode discussed in multimodal RAG guides and in table-heavy failure leaves.

A hybrid is common: OCR plus caption for keyword BM25, native image vectors for visual similarity, fused in hybrid retrieval. Weaviate hybrid search can combine keyword scores with vector scores when captions exist beside image vectors.

Choose with a labelled set that includes questions only answerable from the image pixels. If bridge captions solve those questions, stay on text. If not, pay for native multimodal embeddings.

When should you use multimodal embeddings?

Use multimodal embeddings when users ask about charts, UI screenshots, diagrams or photos whose answer is not fully present in surrounding text. Product manuals, industrial inspection photos and slide decks are the usual corpora in IBM, Hugging Face and arXiv industrial write-ups.

Skip multimodal embeddings when every image is decorative or already fully described in adjacent paragraphs. In that case text RAG is enough and cheaper. Also skip when you lack any labelled image questions to evaluate with.

Architectures that retrieve across modalities at the system level beyond a single embedding space belong on multimodal RAG architectures. This leaf owns the embedding decision.

Why is multimodal retrieval hard to evaluate?

Multimodal retrieval is hard to evaluate because text-only Recall@k ignores whether the correct image was retrieved, caption quality confounds bridge systems, and human relevance for images is slower to label than for passages. Meilisearch and Weaviate both stress modality-aware tests.

Build gold sets with explicit evidence type tags (text, image, table). Score image-hit rate separately. For bridge systems, also score caption faithfulness so you know whether failures are retrieval or captioning.

Public text leaderboards will not save you. Run your own multimodal slice even if it starts at a few dozen questions.

How do you implement multimodal embeddings for RAG?

Implement multimodal embeddings by choosing a multimodal model, embedding images and text with matching towers, storing vectors in Weaviate with metadata for modality, retrieving with the same model at query time, and evaluating on modality-tagged gold questions. Weaviate’s multimodal guide is the default implementation map for this stack.

Keep filenames, OCR text and captions as filterable text properties for hybrid search. Version the multimodal model id beside dimension and preprocessing flags. Re-embed when the model changes; mixed multimodal spaces fail quietly.

Start with a bridge caption baseline if you need a week-one demo, then A/B native multimodal retrieval on the image-only question slice before you delete the bridge.

Publish the language matrix with the index: each active language, the gold-query count, Recall@k, and whether that language uses multilingual embedding or translation. Without the matrix, on-call engineers cannot tell whether a complaint is a model failure or a missing language policy.

When adding a new language, require a minimum gold-set size before marking the language supported in the product UI, so marketing cannot claim coverage the retriever has never been measured on.

Align chunking across languages as far as tokenization allows; wildly different average chunk lengths by language create uneven neighbour quality that looks like an embedding bug.

Retention note 3: keep the prior multilingual checkpoint and its gold-set scores in object storage for ninety days after cutover so a recall regression can be compared against the exact vectors that passed the bake-off rather than against a rebuilt approximation.

Retention note 4: keep the prior multilingual checkpoint and its gold-set scores in object storage for ninety days after cutover so a recall regression can be compared against the exact vectors that passed the bake-off rather than against a rebuilt approximation.

Retention note 5: keep the prior multilingual checkpoint and its gold-set scores in object storage for ninety days after cutover so a recall regression can be compared against the exact vectors that passed the bake-off rather than against a rebuilt approximation.

Retention note 6: keep the prior multilingual checkpoint and its gold-set scores in object storage for ninety days after cutover so a recall regression can be compared against the exact vectors that passed the bake-off rather than against a rebuilt approximation.

Retention note 7: keep the prior multilingual checkpoint and its gold-set scores in object storage for ninety days after cutover so a recall regression can be compared against the exact vectors that passed the bake-off rather than against a rebuilt approximation.

Retention note 8: keep the prior multilingual checkpoint and its gold-set scores in object storage for ninety days after cutover so a recall regression can be compared against the exact vectors that passed the bake-off rather than against a rebuilt approximation.

Retention note 9: keep the prior multilingual checkpoint and its gold-set scores in object storage for ninety days after cutover so a recall regression can be compared against the exact vectors that passed the bake-off rather than against a rebuilt approximation.

Retention note 10: keep the prior multilingual checkpoint and its gold-set scores in object storage for ninety days after cutover so a recall regression can be compared against the exact vectors that passed the bake-off rather than against a rebuilt approximation.

Retention note 11: keep the prior multilingual checkpoint and its gold-set scores in object storage for ninety days after cutover so a recall regression can be compared against the exact vectors that passed the bake-off rather than against a rebuilt approximation.

Retention note 12: keep the prior multilingual checkpoint and its gold-set scores in object storage for ninety days after cutover so a recall regression can be compared against the exact vectors that passed the bake-off rather than against a rebuilt approximation.

Retention note 13: keep the prior multilingual checkpoint and its gold-set scores in object storage for ninety days after cutover so a recall regression can be compared against the exact vectors that passed the bake-off rather than against a rebuilt approximation.

Retention note 14: keep the prior multilingual checkpoint and its gold-set scores in object storage for ninety days after cutover so a recall regression can be compared against the exact vectors that passed the bake-off rather than against a rebuilt approximation.

Retention note 15: keep the prior multilingual checkpoint and its gold-set scores in object storage for ninety days after cutover so a recall regression can be compared against the exact vectors that passed the bake-off rather than against a rebuilt approximation.

Retention note 16: keep the prior multilingual checkpoint and its gold-set scores in object storage for ninety days after cutover so a recall regression can be compared against the exact vectors that passed the bake-off rather than against a rebuilt approximation.

Retention note 17: keep the prior multilingual checkpoint and its gold-set scores in object storage for ninety days after cutover so a recall regression can be compared against the exact vectors that passed the bake-off rather than against a rebuilt approximation.

Retention note 18: keep the prior multilingual checkpoint and its gold-set scores in object storage for ninety days after cutover so a recall regression can be compared against the exact vectors that passed the bake-off rather than against a rebuilt approximation.

Retention note 19: keep the prior multilingual checkpoint and its gold-set scores in object storage for ninety days after cutover so a recall regression can be compared against the exact vectors that passed the bake-off rather than against a rebuilt approximation.

Retention note 20: keep the prior multilingual checkpoint and its gold-set scores in object storage for ninety days after cutover so a recall regression can be compared against the exact vectors that passed the bake-off rather than against a rebuilt approximation.

Retention note 21: keep the prior multilingual checkpoint and its gold-set scores in object storage for ninety days after cutover so a recall regression can be compared against the exact vectors that passed the bake-off rather than against a rebuilt approximation.

Retention note 22: keep the prior multilingual checkpoint and its gold-set scores in object storage for ninety days after cutover so a recall regression can be compared against the exact vectors that passed the bake-off rather than against a rebuilt approximation.

Retention note 23: keep the prior multilingual checkpoint and its gold-set scores in object storage for ninety days after cutover so a recall regression can be compared against the exact vectors that passed the bake-off rather than against a rebuilt approximation.

Retention note 24: keep the prior multilingual checkpoint and its gold-set scores in object storage for ninety days after cutover so a recall regression can be compared against the exact vectors that passed the bake-off rather than against a rebuilt approximation.

Retention note 25: keep the prior multilingual checkpoint and its gold-set scores in object storage for ninety days after cutover so a recall regression can be compared against the exact vectors that passed the bake-off rather than against a rebuilt approximation.

Multimodal embeddings earn their keep when answers live in pixels, not only in prose. Weaviate, IBM, Meilisearch, Elastic, Hugging Face, LangChain and arXiv industrial studies all require modality-aware evaluation. Prefer native shared-space models when captions fail; prefer bridges when OCR already holds the fact. Store vectors in Weaviate with text properties for hybrid exact matches, and never mix multimodal model generations in one index.