Quantization: Cutting Vector Memory Without Cutting Recall

Vector quantization compresses embeddings so an ANN index fits more vectors in memory, trading some distance fidelity for a smaller footprint, and the codec you pick (product, binary or scalar) sets how that trade lands on recall. Weaviate’s PQ, BQ and SQ configuration pages (as of September 2026) define product, binary and scalar quantization as collection-level compression for vector indexes, note that PQ and SQ are not supported on the flat index, and warn that a configuration that reduces memory may also reduce recall. Weaviate’s compression trio sits beside AWS OpenSearch’s quantization techniques post, EnterpriseDB’s vector-quantization overview, Qdrant’s compression writing and Tacnode’s guides (extracted September 2026) covering the same family of methods. Compression sits beside structure choice on what the index does to recall: first pick Flat, HNSW or IVF shape, then decide whether vectors must shrink to fit RAM.

What is vector quantization in a RAG index?

Vector quantization replaces full-precision embedding components with a compact code so distance computations and storage use fewer bits per dimension or per segment. Weaviate’s concepts and compression docs (as of September 2026) treat quantization as a way to reduce an HNSW index’s memory footprint for larger datasets, and state that over-fetching plus rescoring with higher-precision vectors is how many systems recover quality after a compressed first pass. EnterpriseDB’s overview frames the same job as cutting memory and speeding distance math while accepting approximation error.

Quantization is not prompt compression and not Matryoshka dimension truncation. Prompt token budgets live under generation compression. Shrinking embedding width at model time lives under embedding dimensions. Here the object is already a vector in the index; the codec stores a cheaper representation of that vector. Named stores that ship these codecs include Weaviate, Pinecone, Qdrant and Milvus; naming a store is not choosing a codec.

Why do RAG indexes compress vectors?

RAG indexes compress vectors because HNSW and similar in-memory structures grow with vector count times dimensions times bytes, and RAM becomes the ceiling before CPU does. Weaviate’s vector-index concepts page (as of September 2026) states that HNSW stores nodes and edges in memory and points to quantization as the way to reduce that footprint, with disk-oriented HFresh as another escape when memory is the main constraint. AWS OpenSearch’s cost-optimized vector post organises binary, byte, FP16 and product quantization as levers on memory, recall and latency for the same reason.

Without compression, teams either buy larger machines, drop to fewer dimensions (a different trade), or switch index types. With compression, they keep more of the corpus hot and accept that first-stage distances are noisier. Rescoring exists because that noise is expected. The failure mode is treating a vendor’s compression-ratio headline as a recall guarantee on your corpus.

How does product quantization (PQ) compress vectors?

Product quantization splits a vector into segments (subvectors), learns a codebook per segment, and stores each segment as the index of its nearest codeword. Weaviate’s PQ compression documentation (as of September 2026) defines PQ as data compression for vectors that reduces the HNSW index’s memory footprint, states that PQ makes trade-offs among recall, performance and memory, and recommends tuning HNSW together with PQ so the two complement each other. AutoPQ can trigger training from collection size when asynchronous indexing is enabled; the documented default training limit is 100,000 objects per shard, and manual enablement expects a training set of that order before the rest of the data loads.

Weaviate also states that once quantization is set on a collection it cannot be disabled, that default quantization via DEFAULT_QUANTIZATION (from v1.33) is unset unless configured, and that PQ is not applied to unsupported index types such as flat. Distance calculation under PQ uses the codes for the first pass; rescoring pulls fuller vectors for a candidate subset. Segment count and encoder settings change the codebook shape; treat those as product parameters to read from current docs rather than as universal constants.

How does binary quantization (BQ) compress vectors?

Binary quantization maps each dimension (or a transformed coordinate) to a bit so distances can use Hamming-style operations on packed bit vectors. Weaviate’s BQ compression page (as of September 2026) defines BQ as a vector compression technique that reduces vector size, documents enablement on new collections and (from v1.31) on existing ones, and lists rescoreLimit behaviour that differs by index type: under flat, rescoreLimit defaults to -1 (Weaviate picks the limit), while under HNSW a rescoreLimit set for BQ is accepted but silently discarded. Hugging Face’s embedding-quantization writing, cited across practitioner posts, describes float32-to-1-bit conversion as a large memory reduction; that class of “32×” claim is a bit-width identity, not a measured recall result for your RAG corpus.

BQ is attractive when memory pressure is extreme and the embedding geometry tolerates bit codes. It is risky when small angular differences carry the relevance signal. Always pair BQ with a measured rescoring path and a labelled query set. AWS OpenSearch’s binary-quantization section makes the same memory-versus-quality framing for its engine; verify figures on the engine you actually run.

How does scalar quantization (SQ) compress vectors?

Scalar quantization maps each component into a lower-bit integer bucket using learned or fixed boundaries, keeping per-dimension structure rather than segment codebooks. Weaviate’s SQ compression page (as of September 2026) defines SQ as vector compression that reduces vector size, documents enablement on new and (from v1.31) existing collections, and lists parameters including rescoreLimit (default 20 under HNSW, -1 under flat) and trainingLimit for bucket-boundary training. Like PQ, SQ is not supported on the flat index for default-quantization application.

SQ often sits between BQ and full float storage in aggressiveness: more fidelity than 1-bit codes, less compression than the most aggressive PQ setups depending on configuration. EnterpriseDB and AWS write-ups treat byte or scalar-style methods as the “keep more precision” lane. Choose SQ when you need a milder loss than BQ and can afford the training step PQ also needs in spirit if not in identical maths.

Why do quantized indexes over-fetch and rescore?

Quantized indexes over-fetch and rescore because compressed distances rank a noisy candidate list, so the system retrieves more than k compressed neighbours and re-ranks a subset with higher-precision vectors. Weaviate’s quantization concepts (as of September 2026) describe over-fetching and rescoring explicitly, and the BQ/SQ parameter tables expose rescoreLimit as the minimum candidate count before that second pass on supported configurations. Multi-vector embeddings (ColBERT-style) in Weaviate’s notes use compressed vectors in the initial search and uncompressed vectors for MaxSim scoring, the same two-phase idea.

Skipping rescoring to “save latency” is how teams publish a compression win and ship a recall loss. Measure end-to-end recall@k with rescoring on, then decide whether the extra candidates fit the latency budget. If they do not, fix k, rescoreLimit or the codec before blaming the embedder.

How does quantization interact with HNSW and flat indexes?

Quantization interacts with the index type because not every codec is valid on every structure. Weaviate’s September 2026 compression docs state that PQ and SQ are not supported for the flat index under default quantization rules, while BQ can be configured with flat including a meaningful rescoreLimit. HNSW remains the common home for PQ and SQ because that is where memory pressure from full-precision vectors hurts first. Weaviate’s vector-index concepts page also pairs quantization with HNSW memory discussion and names HFresh when the goal is keeping most data on disk rather than compressing an in-memory graph.

Index parameters still matter after compression. Weaviate’s PQ page says a PQ setup that saves memory may reduce recall, and that HNSW should be tuned alongside PQ. Raising query breadth after compression is a common recovery lever; it is not free latency. Depth on graph knobs lives on HNSW; depth on partitions lives on IVF and flat.

How should you choose among PQ, BQ and SQ?

Choose among PQ, BQ and SQ by measuring recall@k and p95 retrieve latency on your own queries at the memory budget you can buy, starting from the mildest codec that fits. Prefer SQ when you want a moderate cut with per-dimension buckets and can train boundaries. Prefer PQ when segment codebooks buy more compression on HNSW and you can run the training limit Weaviate documents (default 100,000 objects per shard for AutoPQ). Prefer BQ when bit packing is the only way the corpus fits and rescoring is mandatory. Prefer no quantization while flat or a modest HNSW already fits RAM.

Refuse to treat a blog’s “32× memory with no recall loss” line as transferable. AWS, EnterpriseDB and vendor docs all show method menus; none of them replaces a labelled evaluation on your chunks. Rotational quantization (RQ) appears in Weaviate’s broader quantization family as of the September 2026 docs set; keep PQ, BQ and SQ as the primary decision set unless your version’s RQ guide is the path you have already validated. When the real problem is permissions or tenants, open filtering and multi-tenancy before you compress. When the real problem is stale deletes, open updates before you retrain a codebook.

Where does rotational quantization fit beside PQ, BQ and SQ?

Rotational quantization appears in Weaviate’s broader quantization family as of the September 2026 docs set, including 8-bit, 4-bit and 1-bit RQ variants on the concepts pages. RQ is another compression path that rotates or transforms coordinates before quantising them, aiming to keep more geometric fidelity for a given bit budget. For day-to-day RAG decisions on this map, validate PQ, BQ and SQ first unless your Weaviate version’s RQ guide is the path you have already measured end to end.

Weaviate’s DEFAULT_QUANTIZATION environment variable (from v1.33) can set a default codec for newly created collections when configured; it is unset by default, meaning no quantization applies unless you opt in. Once quantization is set on a collection, Weaviate’s compression docs state it cannot be disabled. That irreversibility is why a pilot collection should measure recall before you stamp the codec onto every production class.

How does quantization work with named vectors and multi-vectors?

Named vectors let one object carry multiple embeddings, each with its own index configuration. Weaviate’s BQ, PQ and SQ pages (as of September 2026) state that compression must be enabled independently for each named vector, and that every vector can use PQ, BQ, RQ, SQ or no compression. Multi-vector embeddings such as ColBERT-style representations also support those codecs, with compressed vectors used in the initial search and uncompressed vectors used when computing MaxSim for final scoring.

That independence matters in RAG stacks that store a dense retrieval vector beside a sparse or late-interaction representation. Compressing only the large dense space can reclaim RAM while leaving a second vector untouched. Compressing both without measuring MaxSim quality is how multi-vector gains disappear. Keep codec choices per vector name in the collection config review, not as a single global boolean.

When should you prefer a disk-oriented index over heavier quantization?

Prefer a disk-oriented index when even aggressive quantization cannot fit the working set you need hot, or when the product’s HFresh-style design keeps only compressed centroids in memory by intent. Weaviate’s vector-index concepts page (as of September 2026) positions HFresh as a cluster-based index that uses HNSW for the centroid index and keeps most data on disk for memory efficiency, trading some query speed. Quantization and disk-oriented indexes are complementary escapes, not rivals: some deployments compress what stays in RAM and page the rest.

AWS OpenSearch’s disk-based vector search discussion in the live teardown set makes the same qualitative point for that engine: memory, recall and latency move together when vectors leave pure RAM. Refuse invented speed ratios. Measure p95 retrieve latency on your query mix after the architecture change, including cold-cache behaviour if disk is in the path.

How should you measure quantization’s recall cost?

Measure quantization’s recall cost with a labelled query set, an exact or high-recall baseline, and the same k you serve in production, with rescoring configured as it will be in production. Compare full-precision HNSW (or flat on a sample) against the compressed configuration on identical ids. Record recall@k, p50 and p95 latency, and RSS or index size. Weaviate’s PQ guidance to tune HNSW alongside PQ is the product form of “do not change only the codec.”

jkatz’s deep quantization writing and EnterpriseDB’s challenges section both warn that first-stage scores become noisier. Over-fetching is the usual mitigation; under-fetching is the usual silent failure. If recall drops only on filtered queries, the bug may be filter interaction rather than the codec. If recall drops only after a bulk load that skipped PQ training limits, the bug is training, not bits.

Training limits are safety rails, not superstition. Weaviate’s PQ docs (as of September 2026) tell operators to wait for training completion logs before loading the rest of a manually configured collection, and AutoPQ waits until the object count reaches the training limit (default 100,000 per shard). Loading everything before the codebook exists produces a different failure than picking the wrong codec: the index may run uncompressed or half-trained depending on timing. Read the system logs the docs point to before you declare compression “on.”

Encoder choice inside PQ (for example tile encoders with a distribution setting in client examples) changes how segment codebooks are built. Treat encoder parameters as versioned product settings: copy them from the Weaviate PQ parameter table for your release, then re-measure recall. Read Weaviate’s own PQ parameter list before importing bit allocations from any FAISS IVFPQ notebook.

Binary quantization’s silent discard of rescoreLimit under HNSW in Weaviate’s BQ parameter table (as of September 2026) is a sharp edge: an API accepts the field, but reading the collection definition back will not show it, and HNSW BQ will not use that limit the way flat BQ does. Configure rescoring expectations per index type. Flat plus BQ with an explicit rescore limit is a different machine from HNSW plus BQ.

Scalar quantization’s default rescoreLimit of 20 under HNSW is a starting point, not a law. Raise it when compressed first-stage rankings scramble the true top-k; lower it only after measuring that recall holds. Pair every change with the same labelled queries you used to accept the codec.

Cost-optimized vector platforms such as AWS OpenSearch publish comparative tables across binary, byte, FP16 and product methods. Use those tables as hypotheses for that engine, then re-run the same comparison shape on Weaviate PQ, BQ and SQ with your embeddings. Cross-engine compression ratios are not portable because rescoring defaults and index types differ.

MongoDB and Medium teardown pages in the live set often summarise compression at a high level without index-type constraints. Prefer Weaviate’s PQ/BQ/SQ pages when the serving database is Weaviate, and prefer the engine you actually run otherwise. The gain on this leaf is the constraint set: flat limits, training limits, rescoring and irreversibility after enablement.

Rescoring increases candidate traffic. Budget network and CPU for the second pass when you raise rescoreLimit, especially on flat indexes where BQ rescoring is explicit. A memory win that doubles query CPU is still a production change you must accept deliberately.

After enablement, watch RSS and recall together for a full traffic week. Compression that looks fine on a lab sample can still scramble long-tail queries that matter to support teams.

Quantization is the memory dial on top of an index structure, not a substitute for choosing Flat, HNSW or IVF. Weaviate’s PQ, BQ and SQ pages (as of September 2026) make the constraints explicit: collection-level enablement, training limits for PQ, rescoring parameters, and flat-index limits for PQ and SQ. Compress only after you know the structure, measure recall with rescoring on, and treat every compression-ratio headline as a hypothesis until your query set agrees.

More in Indexing for RAG: Structures, Trade-offs and Limits Toggle section list