Sparse Embeddings: SPLADE and Learned Lexical Retrieval
The middle ground between BM25 and dense vectors, and when it outperforms both.
Sparse embeddings are high-dimensional vectors where most entries are zero; each non-zero dimension is a term/subword weight. Classic sparse retrieval (like BM25) keeps exact term matching, while learned sparse models such as SPLADE add expansion weights that BM25 can’t invent—often improving recall on rare keywords and vocabulary mismatch cases.
How does SPLADE generate sparse embeddings?
SPLADE generates sparse vectors by projecting a BERT masked-language-model (MLM) head into the full WordPiece vocabulary, then aggregating token contributions into a sparse bag of weighted terms.
The mechanism has two parts:
- WordPiece → vocabulary weights. SPLADE reuses BERT’s MLM-style head so each input token produces a distribution over the full vocabulary (about 30,522 WordPiece entries) and those logits become per-term importance signals (Naver Labs / SPLADE writeups; Pinecone SPLADE tutorial).
- Aggregation + sparsity control. SPLADE applies a log-saturated / ReLU-based aggregation (sum or max pooling depending on variant) to build a single sparse vector per input, and a FLOPS-regularizer to keep the number of active terms small enough for inverted-index search (Paria et al. FLOPS regularizer; Naver Labs, SIGIR 2021).
SPLADE also learns expansion: it can assign weight to terms that were not explicitly present in the surface string, reducing the vocabulary mismatch between query and relevant passages (Pinecone tutorial; Qdrant sparse-vectors overview).
How do sparse embeddings differ from BM25?
BM25 weights only observed terms using corpus statistics, while learned sparse embeddings predict term importance and can expand queries/documents into related vocabulary.
In other words:
- BM25 is frequency/statistics-based. It boosts terms that appear more often and compensates for length, but it does not generate new vocabulary connections.
- SPLADE is neural and context-conditioned. It uses a pretrained language model to decide which terms matter and how to activate them from context, including expansions that address vocabulary mismatch (HF sparse-retrieval survey framing; Zilliz BM25 vs SPLADE comparison).
- Trade-off. BM25 is typically faster and simpler; SPLADE can be slower because it often has more non-zero postings.
When should you use sparse embeddings instead of dense?
You should use sparse embeddings (learned sparse included) when your queries depend on rare keywords, identifiers, or domain vocabulary where paraphrases alone still miss relevant passages.
For intuition, Qdrant’s SPLADE overview reports MS MARCO Dev retrieval numbers (MRR@10): BM25 at 0.184, SPLADE at 0.322, SPLADE-max at 0.340, and DistilSPLADE-max at 0.368 (Qdrant, 2023).
Use dense embeddings instead when:
- your corpus is mostly text with enough lexical overlap, and
- your users search with paraphrases rather than exact identifiers.
And when the workload mixes both “exact token” and “semantic paraphrase”, sparse+dense hybrid is usually the next step; fusion mechanics live on Hybrid Search.
Should you combine sparse and dense embeddings?
Most production stacks combine sparse and dense retrieval because each channel covers different failure modes.
The goal is to retrieve candidates that sparse misses (semantic paraphrase) and candidates that dense misses (exact identifiers and exact terms). This page focuses on the sparse embedding mechanism; fusion and the measured hybrid gains live on Hybrid Search.
What does SPLADE cost in latency and memory?
SPLADE is often slower than classic BM25 because its sparse vectors can have more active terms, which increases inverted-index work.
Two sources of cost show up in the SPLADE ecosystem:
- More non-zeros → bigger postings lists. Naver Labs describes the FLOPS-regularizer as an explicit efficiency knob: with strong regularization, a “far left” SPLADE configuration produces representations that average about 18 terms per passage, while MS MARCO passages average around 60 terms (Naver Labs, 2021).
- Neural forward-pass overhead. SPLADE requires running a BERT-like encoder to compute sparse weights, whereas BM25 is a tokenize-and-score baseline.
On memory scale, Qdrant’s worked example (under its stated assumptions) sketches the shape of the benefit: dense embeddings at 6.144 GB versus sparse at 1.12 GB for a 1M-document scenario (Qdrant, 2023).
| Vector Type | Memory (GB) |
|---|---|
| Dense BERT vector | 6.144 |
| OpenAI embedding | 12.288 |
| Sparse vector | 1.12 |
Measure before committing
Both latency and memory depend on vector sparsity, your inverted-index implementation, and your ANN/keyword pipeline. SPLADE’s FLOPS knob is there specifically so you can trade effectiveness for efficiency in a controlled way.
How do you implement sparse embeddings for RAG?
You implement sparse embeddings for RAG by encoding documents and queries into sparse {term_id:weight} vectors, storing them in a vector store that supports sparse vectors, and then retrieving with sparse scoring (optionally fused with dense).
The implementation handoff:
- Encode corpus. Generate sparse term-weight vectors for each document.
- Store with sparse-vector support. Use a vector database / engine that can store and query sparse vectors (Qdrant, Pinecone, and others; Weaviate supports sparse-vector modes in its retrieval guidance).
- Encode queries with the same model. Produce query sparse vectors in the same vocabulary space.
- Optional hybrid. If your system needs both exact tokens and semantic recall, route fusion to Hybrid Search.
For a runnable, pinned pipeline template, see building the pipeline.
What are sparse embeddings?
Sparse embeddings are vectors where most entries are zero and each non-zero dimension corresponds to a term/subword weight. This keeps exact-term matching and interpretability, and learned sparse models such as SPLADE can also add expansion weights for related terms.
How does SPLADE work?
SPLADE runs a BERT-like encoder with an MLM head, projects token representations into a full WordPiece vocabulary (around 30,522 entries), then aggregates those logits into a sparse term-weight vector. FLOPS-style regularization controls how many terms stay active, and the model learns query/document expansion so it can activate terms not explicitly present in the surface text.
Sparse vs BM25 — which first?
Start with BM25 when you need a fast, interpretable lexical baseline and your queries already overlap with corpus tokens. Move to learned sparse embeddings like SPLADE when BM25 struggles with vocabulary mismatch and rare terms—Qdrant’s SPLADEv2 overview reports higher MS MARCO dev MRR@10 for SPLADE variants than BM25.
Sparse vs dense — when each?
Use sparse embeddings when your retrieval target depends on rare keywords, identifiers, and exact terminology, and when you want term-weight interpretability. Use dense embeddings when users rely on paraphrase/semantic similarity and exact term overlap is weak.
Do I still need hybrid if I use SPLADE?
Often yes. SPLADE improves the lexical channel, but dense retrieval still covers semantic paraphrase cases. Hybrid retrieval exists to cover both failure modes together, and fusion mechanics (RRF/alpha trade-offs and measured gains) are handled on the Hybrid Search page.