Quantization: Cutting Vector Memory Without Cutting Recall

Vector quantization compresses embedding vectors so the index stores fewer bits per dimension (or a short code) instead of full float32. The benefit is lower RAM and disk footprint; the cost is approximate distance math that can reduce recall unless you tune and often re-score.

How does scalar quantization work?

Scalar quantization (SQ) maps each float32 dimension independently into an 8-bit integer bucket, keeping the embedding shape the same but reducing precision. In Weaviate’s SQ documentation, SQ turns float32 into an 8-bit integer (an approximately 4× size reduction) by distributing values into 256 buckets.

Scalar quantization is mostly a calibration-and-rounding process. It works like this:

1. Analyze values on a training/calibration set. Weaviate’s SQ approach uses data to decide how to map float ranges into buckets.
2. Define the value range and bucket boundaries. The algorithm derives bucket intervals from the training distribution.
3. Convert float values to int8 bucket IDs. Each dimension becomes one 8-bit code.
4. Run similarity on the compressed representation. The system computes approximate distances on the int8 values (and can optionally re-score with full-precision vectors).

How does product quantization work?

Product quantization (PQ) reduces memory by splitting a high-dimensional vector into segments/subspaces, training a small codebook for each segment, and then storing one centroid ID per segment. Weaviate describes PQ as two-stage compression: reduce dimensionality into “segments”, then quantize each segment and build a codebook of centroids (defaulting to 256 centroids per segment).

PQ’s structural intuition is byte math. In Weaviate’s PQ example for 768 four-byte elements, PQ changes storage from 768 × 4 = 3072 bytes to 128 × 1 = 128 bytes, making the vector representation “almost 24 times” smaller (with a small codebook overhead).

How does binary quantization work?

Binary quantization (BQ) converts each embedding dimension into a 1-bit representation (often using a threshold at 0 for normalized embeddings), enabling Hamming-style distance on packed bits. Hugging Face’s embedding-quantization blog describes BQ as a conversion of float32 values to 1-bit values, yielding a 32× reduction in memory and disk usage.

BQ is usually paired with re-scoring to recover ranking quality. Hugging Face reports that for mxbai-embed-large-v1 on MTEB Retrieval, binary embeddings retain 92.53% of retrieval performance without rescoring, and rescoring lifts this to 96.45%.

How much memory does quantization save?

Quantization saves memory by changing how many bits you store per dimension (or by replacing floats with compact codes). The exact end-to-end memory often differs from the raw bit-level ratio because vector DBs also store index structures, graphs, and metadata.

The key pattern is consistent across implementations: SQ and BQ change precision directly, while PQ also introduces a learned codebook and segmenting step. Also remember that graph-based ANN indexes (for example HNSW) keep additional structures that can limit “full” raw savings.

Does quantization reduce recall?

Quantization is lossy, so quantization can reduce recall@k compared to float32—sometimes slightly (SQ), and sometimes more noticeably (BQ) unless you re-score or over-fetch enough candidates.

When you do measure, you usually see the same cause-and-effect: compression error changes distances, so the approximate nearest neighbors returned by the compressed index can miss some true top-k items. Weaviate’s quantization docs also describe the common mitigation: over-fetch compressed candidates and re-score with the stored original vectors.

How do oversampling and rescoring recover recall?

Oversampling and rescoring recover lost recall by decoupling “fast candidate retrieval” from “final ranking.” Weaviate’s vector quantization docs describe that when SQ, RQ, or BQ is enabled, Weaviate boosts recall by over-fetching compressed results and then comparing the corresponding original vectors for re-scoring.

A representative shape is:

• Fetch more candidates than you need. Weaviate’s example uses a limit of 10 and a rescore limit of 200.
• Re-score the shortlist with full-precision vectors. The system re-computes distances so compression noise does not decide the final order.

The trade-off is compute and memory pressure: more candidates increases query-time latency, and keeping original vectors around consumes extra storage. Without a large enough candidate pool, rescoring cannot recover neighbors that were never retrieved.

Which quantization should you use for RAG?

Choose quantization based on the RAG workload’s memory pressure and your ability to re-score, not based on a single “compression ratio” headline. A practical decision ladder is:

1. Mild memory pressure / first pass: start with scalar quantization (int8 / SQ) and enable oversampling + re-scoring so ranking quality stays close to float32.
2. Extreme scale / hard RAM ceiling: consider binary quantization (BQ), but assume you will need rescoring (Hugging Face reports rescoring lifts binary performance for mxbai-embed-large-v1).
3. Aggressive compression with controlled training: consider product quantization (PQ) when you can train a codebook (Weaviate recommends a training set size per shard).

Across these choices, Weaviate points out that an index type may constrain which quantizations are supported (for example, PQ and SQ aren’t supported for the flat index). If your pipeline cannot keep originals for re-scoring, prefer the gentler compression and measure first.

How does quantization work with HNSW?

Quantization changes what the HNSW index stores and how distance is computed—so you tune HNSW parameters together with the compression settings. Weaviate’s vector quantization documentation explains that an HNSW index can be configured using PQ, SQ, RQ, or BQ, and compression can reduce memory footprint or allow you to store more data in the same amount of memory.

One practical co-tuning implication is simple: if compression reduces distance fidelity, you typically need a query-time setting that compensates (for example, by exploring more candidates). Weaviate also describes that for SQ, RQ, or BQ, Weaviate re-scores results using original uncompressed vectors, which is one of the ways it preserves retrieval quality.

How do you enable quantization in a vector database?

Vector DBs enable quantization by configuring compression on the collection/index—so the index stores compressed codes and optionally keeps original vectors for re-scoring. Weaviate’s documentation describes enabling SQ, PQ, RQ, and BQ via vector-quantization concepts and configuration references.

For a runnable, pinned configuration, this site routes you to building the pipeline (mechanism pages explain the “what it costs and why it works,” not full SDK walkthroughs). After you enable compression, measure the before/after impact with retrieval metrics on your own query set.