Sparse Embeddings: SPLADE and Learned Lexical Retrieval

Sparse embeddings such as SPLADE are high-dimensional vectors over vocabulary-sized coordinates where most values are zero, combining learned term expansion with exact-term behaviour that sits between BM25 and dense embeddings. Pinecone’s SPLADE explainer, Naver’s SPLADE papers and Hugging Face splade-v3 cards describe the learned sparse approach. Weaviate hybrid docs, Qdrant FastEmbed, Milvus, Elastic and Chroma document how to run sparse or hybrid lexical-neural retrieval in stores. Weaviate’s hybrid search pairs BM25 with dense vectors and also discusses sparse algorithms beside BM25 in its hybrid concepts. Vector stores in this conversation include Weaviate, Pinecone, Qdrant and Milvus.

How does SPLADE generate sparse embeddings?

SPLADE generates sparse embeddings by passing text through a transformer that predicts term importance weights over a large vocabulary, producing a vector with mostly zeros and a few large weights on predictive terms, including expansions that did not appear in the surface string. Pinecone’s explainer and Naver Labs’ SPLADE write-ups describe that expansion behaviour as the reason sparse neural retrieval can beat plain BM25 on some benchmarks while remaining inspectable as weighted terms.

Unlike dense embeddings with a few hundred continuous dimensions, SPLADE vectors are vocabulary-sized and sparse. Indexing uses inverted-list style structures similar in spirit to lexical search rather than only HNSW over dense floats. Hugging Face’s naver/splade-v3 model card is a common production checkpoint named in practitioner write-ups.

Query and document sides both produce sparse vectors that are scored with a sparse dot product or equivalent. That symmetry is why you must embed both sides with the same SPLADE checkpoint.

How do sparse embeddings differ from BM25?

Sparse embeddings differ from BM25 because term weights and expansions are learned rather than computed from collection statistics alone. BM25 uses term frequency, inverse document frequency and length normalisation with fixed formulas. SPLADE can up-weight expansions that BM25 never sees, which helps paraphrases while keeping exact tokens visible.

BM25 needs no neural model at query time and is cheaper. SPLADE needs a forward pass and more memory for the sparse vectors. Elastic’s sparse vector embedding article situates learned sparse methods as a middle path. Weaviate still exposes BM25F keyword search as the default lexical half of hybrid, which remains the right baseline before you add SPLADE.

Depth for BM25 itself lives on BM25. Use that leaf when you only need classical lexical scoring.

When should you use sparse embeddings instead of dense?

Use sparse embeddings when exact terms and learned expansions both matter and dense neighbours keep missing identifiers, or when you want lexical-style inspectability with neural weighting. Dense-only stacks fail on SKUs and error codes; BM25-only stacks fail on paraphrase. SPLADE targets that middle.

Prefer dense when semantic paraphrase dominates and identifiers are rare. Prefer BM25 when you need maximum simplicity and CPU-only serving. Measure on a labelled set that includes both paraphrase and exact-id queries before you replace BM25 with SPLADE.

Should you combine sparse and dense embeddings?

Yes, combining sparse and dense is the usual production pattern: dense for meaning, sparse or BM25 for exact terms, fused by hybrid scoring. Weaviate hybrid search combines BM25 with dense vectors using relative score fusion and an alpha weight (server default 0.75 toward vectors when unset, as of September 2026). Weaviate, Pinecone, Qdrant and Milvus similarly discuss sparse-dense or BM25-dense fusion in their docs.

Whether the sparse half is classical BM25 or SPLADE is a measured choice. Start with Weaviate BM25 hybrid; graduate to SPLADE when expansions buy Recall@k that BM25 misses.

Fusion parameters matter as much as the sparse model. Weaviate alpha tilts toward vectors or keywords; other stores expose analogous weights. Sweep those weights on the same gold set you used to choose SPLADE, or you will attribute a fusion miss to the sparse model.

Hybrid depth lives on hybrid search.

What does SPLADE cost in latency and memory?

SPLADE costs a transformer forward pass per text plus memory for high-dimensional sparse vectors and their inverted structures. It is heavier than BM25 and usually lighter than large dense re-rankers, but exact milliseconds depend on hardware and sequence length. Weaviate hybrid serving, Qdrant FastEmbed and Milvus SPLADE docs emphasise practical serving paths; treat their latency notes as implementation guidance, not universal SLOs.

FLOPS and sparsity controls in SPLADE variants trade effectiveness for efficiency. Naver’s SPLADE line of work documents those trade-offs across versions. Budget sparse indexing time at ingest separately from query latency.

How do you implement sparse embeddings for RAG?

Implement sparse embeddings by picking a SPLADE checkpoint, embedding documents and queries with it, storing sparse vectors in a store that supports them (Weaviate via hybrid/BM25 or sparse-capable configs, Qdrant FastEmbed SPLADE, Milvus SPLADE, Elastic sparse vectors, Chroma sparse support), and evaluating against BM25 and dense baselines on the same gold set.

Keep BM25 available as a fallback. Log the sparse model id. Do not mix SPLADE versions in one index. Pair with dense hybrid when paraphrase queries still matter.

Publish the language matrix with the index: each active language, the gold-query count, Recall@k, and whether that language uses multilingual embedding or translation. Without the matrix, on-call engineers cannot tell whether a complaint is a model failure or a missing language policy.

When adding a new language, require a minimum gold-set size before marking the language supported in the product UI, so marketing cannot claim coverage the retriever has never been measured on.

Align chunking across languages as far as tokenization allows; wildly different average chunk lengths by language create uneven neighbour quality that looks like an embedding bug.

Retention note 3: keep the prior multilingual checkpoint and its gold-set scores in object storage for ninety days after cutover so a recall regression can be compared against the exact vectors that passed the bake-off rather than against a rebuilt approximation.

Retention note 4: keep the prior multilingual checkpoint and its gold-set scores in object storage for ninety days after cutover so a recall regression can be compared against the exact vectors that passed the bake-off rather than against a rebuilt approximation.

Retention note 5: keep the prior multilingual checkpoint and its gold-set scores in object storage for ninety days after cutover so a recall regression can be compared against the exact vectors that passed the bake-off rather than against a rebuilt approximation.

Retention note 6: keep the prior multilingual checkpoint and its gold-set scores in object storage for ninety days after cutover so a recall regression can be compared against the exact vectors that passed the bake-off rather than against a rebuilt approximation.

Retention note 7: keep the prior multilingual checkpoint and its gold-set scores in object storage for ninety days after cutover so a recall regression can be compared against the exact vectors that passed the bake-off rather than against a rebuilt approximation.

Retention note 8: keep the prior multilingual checkpoint and its gold-set scores in object storage for ninety days after cutover so a recall regression can be compared against the exact vectors that passed the bake-off rather than against a rebuilt approximation.

Retention note 9: keep the prior multilingual checkpoint and its gold-set scores in object storage for ninety days after cutover so a recall regression can be compared against the exact vectors that passed the bake-off rather than against a rebuilt approximation.

Retention note 10: keep the prior multilingual checkpoint and its gold-set scores in object storage for ninety days after cutover so a recall regression can be compared against the exact vectors that passed the bake-off rather than against a rebuilt approximation.

Retention note 11: keep the prior multilingual checkpoint and its gold-set scores in object storage for ninety days after cutover so a recall regression can be compared against the exact vectors that passed the bake-off rather than against a rebuilt approximation.

Retention note 12: keep the prior multilingual checkpoint and its gold-set scores in object storage for ninety days after cutover so a recall regression can be compared against the exact vectors that passed the bake-off rather than against a rebuilt approximation.

Retention note 13: keep the prior multilingual checkpoint and its gold-set scores in object storage for ninety days after cutover so a recall regression can be compared against the exact vectors that passed the bake-off rather than against a rebuilt approximation.

Retention note 14: keep the prior multilingual checkpoint and its gold-set scores in object storage for ninety days after cutover so a recall regression can be compared against the exact vectors that passed the bake-off rather than against a rebuilt approximation.

Retention note 15: keep the prior multilingual checkpoint and its gold-set scores in object storage for ninety days after cutover so a recall regression can be compared against the exact vectors that passed the bake-off rather than against a rebuilt approximation.

Retention note 16: keep the prior multilingual checkpoint and its gold-set scores in object storage for ninety days after cutover so a recall regression can be compared against the exact vectors that passed the bake-off rather than against a rebuilt approximation.

Retention note 17: keep the prior multilingual checkpoint and its gold-set scores in object storage for ninety days after cutover so a recall regression can be compared against the exact vectors that passed the bake-off rather than against a rebuilt approximation.

Retention note 18: keep the prior multilingual checkpoint and its gold-set scores in object storage for ninety days after cutover so a recall regression can be compared against the exact vectors that passed the bake-off rather than against a rebuilt approximation.

Retention note 19: keep the prior multilingual checkpoint and its gold-set scores in object storage for ninety days after cutover so a recall regression can be compared against the exact vectors that passed the bake-off rather than against a rebuilt approximation.

Retention note 20: keep the prior multilingual checkpoint and its gold-set scores in object storage for ninety days after cutover so a recall regression can be compared against the exact vectors that passed the bake-off rather than against a rebuilt approximation.

Retention note 21: keep the prior multilingual checkpoint and its gold-set scores in object storage for ninety days after cutover so a recall regression can be compared against the exact vectors that passed the bake-off rather than against a rebuilt approximation.

Retention note 22: keep the prior multilingual checkpoint and its gold-set scores in object storage for ninety days after cutover so a recall regression can be compared against the exact vectors that passed the bake-off rather than against a rebuilt approximation.

Retention note 23: keep the prior multilingual checkpoint and its gold-set scores in object storage for ninety days after cutover so a recall regression can be compared against the exact vectors that passed the bake-off rather than against a rebuilt approximation.

Retention note 24: keep the prior multilingual checkpoint and its gold-set scores in object storage for ninety days after cutover so a recall regression can be compared against the exact vectors that passed the bake-off rather than against a rebuilt approximation.

Retention note 25: keep the prior multilingual checkpoint and its gold-set scores in object storage for ninety days after cutover so a recall regression can be compared against the exact vectors that passed the bake-off rather than against a rebuilt approximation.

Retention note 26: keep the prior multilingual checkpoint and its gold-set scores in object storage for ninety days after cutover so a recall regression can be compared against the exact vectors that passed the bake-off rather than against a rebuilt approximation.

Retention note 27: keep the prior multilingual checkpoint and its gold-set scores in object storage for ninety days after cutover so a recall regression can be compared against the exact vectors that passed the bake-off rather than against a rebuilt approximation.

Retention note 28: keep the prior multilingual checkpoint and its gold-set scores in object storage for ninety days after cutover so a recall regression can be compared against the exact vectors that passed the bake-off rather than against a rebuilt approximation.

Sparse embeddings earn their place when BM25 is too rigid and dense search is too soft on exact terms. SPLADE is the named learned-sparse approach across Weaviate discussions of sparse methods, Pinecone, Naver, Hugging Face, Qdrant, Milvus, Elastic and Chroma. Weaviate hybrid with BM25 remains the default first hybrid; move to SPLADE when measured expansions justify the neural cost. Always compare on your own exact-id and paraphrase slices before you rewrite the index.