Embedding Dimensions, Truncation and Matryoshka

Embedding dimensions are the length of the dense vector each chunk or query becomes in RAG, and that length sets both the capacity for semantic nuance and the linear bill for storage, memory and similarity-search compute. Particula Tech’s production note on 384 versus 3072 dimensions states the common mistake: teams default to the highest width their model offers and assume bigger is better, while dropping from 1536 to 384 dimensions has cut query latency in half and reduced vector-database costs by about 75 percent in systems they audited, with no measurable retrieval-accuracy drop on those workloads. Microsoft Learn’s Azure generate-embeddings phase treats dimensionality reduction as a first-class cost control after model choice. Kusupati et al.’s Matryoshka Representation Learning (arXiv 2205.13147, 2022) is the training idea that makes prefix truncation credible, and OpenAI’s text-embedding-3 family exposes a dimensions parameter that shortens vectors the Matryoshka way (OpenAI, 2024). Vector stores that hold those vectors include Weaviate, Pinecone, Qdrant and Milvus. What follows covers the cost equation, Matryoshka, safe truncation, when to shrink, what happens on a dimension mismatch, how to choose a width, and how to implement truncation without mixing spaces.

How do embedding dimensions affect storage and retrieval cost?

Embedding dimensions affect storage and retrieval cost mostly linearly because you store and compare one numeric value per dimension for every vector in the index. The structural storage model is bytes approximately equal to vector_count times dimensions times bytes_per_value. For float32 embeddings at 4 bytes per value, one million vectors need about 1.5 GB at 384 dimensions, about 6.1 GB at 1536 dimensions, and about 12.3 GB at 3072 dimensions. That 6.1 GB figure for one million 1536-dimension float32 vectors is the same order of magnitude called out in arXiv 2505.00105 (May 2025) on embedding-storage optimisation for RAG, which also notes that indexes and overhead sit on top of the raw vector bytes.

Particula’s managed-store illustration for ten million vectors puts monthly cost on the same linear track: about 3.75 US dollars per month at 384 dimensions versus about 30 US dollars per month at 3072 dimensions in their example pricing, a reminder that width is a bill you pay every month, not only at ingest. Milvus’s CCKM write-up makes the same point at one hundred million float32 vectors: going from 3072 to 256 dimensions is a 12x storage cut, illustrated as roughly 1.14 TB down to 95 GB for that collection size. Query latency follows the same direction: every candidate-versus-query comparison does more arithmetic as dimensions rise, and HNSW-style indexes need more memory for wider vectors.

GMI Cloud’s production RAG infrastructure note places the same linear pressure on GPU and vector-search capacity: wider embeddings raise both the bytes moved per query and the working set that must stay hot for low-latency ANN. Beam’s embedding-model guide and GreenNode’s dimensionality criterion repeat the trade-off in selection language: vector width is a cost and latency axis beside accuracy, not a vanity metric. Galileo’s selection article likewise ties vector dimension to storage and compare-time before anyone opens a leaderboard.

Quantization is a complementary lever, not a substitute for choosing width. ArXiv 2505.00105 reviews int8 as about a 4x compression versus float32 and binary quantization as about a 32x cut with a re-scoring step to recover quality, citing Shakir et al. (2024) recovering from about 92.5 percent to about 96 percent of baseline after re-rank in their experiments. Combining fewer dimensions with lower precision compounds savings: their figure shows 1536-dimension float32 at about 6 KB per vector shrinking to about 0.37 KB at 384 dimensions and 8-bit values, a 16x example. Exact latency and dollar numbers still depend on your store, ANN parameters and hardware, so measure end-to-end on your own setup after you pick a candidate width. The next mechanism that makes shorter vectors credible without training a separate small model is Matryoshka Representation Learning.

What is Matryoshka Representation Learning?

Matryoshka Representation Learning trains embeddings so that prefixes of the vector stay useful when you truncate the tail, like nesting dolls of decreasing size. Kusupati et al. introduced MRL in 2022 (arXiv 2205.13147): training applies losses across multiple nested widths so earlier dimensions carry the strongest ranking signal. Sentence-Transformers documents the idea as MatryoshkaLoss for practitioners who train their own retrieval embedders. OpenAI’s text-embedding-3 family uses a related shortening capability and points back to that paper when documenting the dimensions parameter (OpenAI, 2024).

Milvus’s CCKM suite treats MRL truncation as a first-class production axis rather than a blog footnote. They generated full-dimension embeddings for 150 STS-B sentence pairs with human similarity scores from 0 to 5, then truncated to 1024, 512 and 256 dimensions and compared Spearman correlation. In that test, Voyage Multimodal 3.5 and Jina Embeddings v4 lost less than 1 percent quality at 256 dimensions, while other models degraded more; Gemini Embed 2 was strong at full width and weaker under truncation in their table. The lesson is not that every model can be cut to 256 safely: it is that MRL-trained or Matryoshka-aware models make the cut measurable, and non-MRL slicing can discard the wrong coordinates.

Practically, MRL changes how you plan capacity. Instead of training or buying three separate models at 256, 512 and 1536 dimensions, you train or buy one Matryoshka-aware model and pick the prefix that clears your recall bar. That is why OpenAI’s dimensions parameter and Sentence-Transformers MatryoshkaLoss show up in production runbooks: they turn width into a configuration choice with a measured quality curve rather than a second procurement cycle.

MRL is different from post-hoc PCA or random projection. ArXiv 2505.00105 contrasts MRL’s train-time nested objectives with post-hoc PCA, Kernel PCA, UMAP, autoencoders and random projections applied after a single high-dimensional model exists. PCA can still help pick a width, as Wang (2019) argued for selecting dimensionality from a trained high-dimensional model, but it is not the same contract as a provider dimensions API that returns an already-ordered prefix. Once you know MRL exists, the operational question is how to truncate without breaking retrieval.

How do you truncate embedding dimensions safely?

You truncate embedding dimensions safely by using the provider’s shortening API for Matryoshka-trained models, re-normalising if you slice manually, keeping the same width at index time and query time, and verifying recall on your labelled set before you cut production. Prefer the provider parameter when it exists: OpenAI embeddings let you pass dimensions to request a shorter output directly rather than shortening after the fact (OpenAI, 2024). OpenAI also reports that shortening text-embedding-3-large to 256 dimensions still outperformed the unshortened ada-002 baseline at 1536 dimensions on MTEB in their release numbers, which is a dated vendor claim to re-check, not a universal law.

If you change dimensions by slicing vectors yourself, re-apply L2 normalisation before storing and searching. OpenAI’s embeddings guide, Voyage documentation and Sentence-Transformers usage notes all warn that truncated vectors can leave the unit sphere and distort cosine ranking if you skip that step. Use MRL-aware models when you plan Matryoshka-style truncation; arbitrary slicing of a non-MRL model can degrade quality sharply even when the vectors remain the “right” length. Milvus notes that truncated MRL vectors are still ordinary float vectors: create the collection with the truncated dimension and insert the shortened vectors, rather than storing 3072 floats and hoping the store ignores the tail.

Safe truncation also means matching preprocessing. If the full-width pipeline lowercased, stripped HTML or capped tokens at 512, the truncated pipeline must do the same, or you will attribute a recall drop to dimensions when the real change was text normalisation. Keep a golden query set that includes exact identifiers, paraphrases and the hard near-miss pairs Particula describes for technical docs, because those are the queries that reveal when a prefix has discarded the distinguishing coordinates.

Do not mix truncation styles or widths in one collection. A 1536-dimension index queried with a 384-dimension vector is a shape error or a silent wrong-space search depending on the client. Changing width is a migration: re-embed the corpus, or dual-write during cutover, then flip traffic. That migration cost is why truncation belongs in the same conversation as embedding drift and why the next question is when shrinking is worth doing at all.

When should you shrink embedding dimensions?

You should shrink embedding dimensions when storage, memory or query latency dominate and a measured recall drop on your labelled queries is still acceptable. Particula’s production framing is blunt: for most business RAG systems, 384 to 768 dimensions deliver the best balance of accuracy, speed and cost, and higher widths (1024 and above) mainly help for complex technical docs, multilingual content or fine-grained semantic distinctions. Their e-commerce example reports about 94 percent retrieval accuracy at 384 dimensions versus about 96 percent at 1024 dimensions, rarely worth a 3x cost increase on that workload. Latency-sensitive agent-assist features in their write-up moved under about 30 milliseconds after switching from 1536 to 384 dimensions.

Particula’s million-document storage example is worth repeating in planning docs: about 12 GB of vector storage at 3072 dimensions versus about 1.5 GB at 384 dimensions for the same count, before ANN structures. That gap shows up as RAM for in-memory indexes, as disk for on-disk segments, and as network bytes when replicas rebuild. Teams that treat dimensions as a free accuracy upgrade discover the bill when the third replica comes online.

Keep higher dimensions when evaluation says you need them. Particula lists complex technical documentation with overlapping terminology, multi-domain enterprise corpora that falsely mix legal and marketing neighbours, near-duplicate semantic distinctions, and multilingual or cross-lingual search as the cases where 1024-plus widths often pay off. Their semiconductor example: 384 dimensions could not separate “voltage tolerance threshold” from “voltage threshold tolerance,” and 1024 dimensions resolved the ambiguity. Azure’s generate-embeddings guidance likewise says larger models are not always better and pairs dimensionality reduction with retrieval evaluation to find the balance.

Shrink decisions should be written as a threshold, not a vibe. Example thresholds used in practice: accept a shrink when labelled recall at the prompt’s top-k falls by less than one or two absolute points while storage or p95 query latency improves by a double-digit percent. Reject a shrink when the failures concentrate on the exact class of queries your users escalate, such as regulatory clause lookup or SKU search, even if the average metric still looks fine. Particula’s “almost right but not quite” complaint is often that concentrated failure mode.

Start narrow when you are bootstrapping. Particula advises beginning with lower dimensions so you can iterate cheaply, then scale up only when measured accuracy gaps appear. That order matches Weaviate’s embedding-model guidance to start with a small baseline and upgrade when the pipeline already works. Shrinking after you already indexed at full width still means a full re-embed, so the cheap path is to measure early. When the geometry is wrong in a different way, the failure is not width but a mismatched pair of spaces.

What happens if embedding dimensions do not match?

If embedding dimensions do not match between the indexed vectors and the query vector, the store either rejects the query or compares incompatible shapes and retrieval becomes undefined for that collection. Clients and servers that validate shape will error; systems that coerce or pad silently produce rankings that are not cosine neighbours in the intended space. The same-model rule from Azure’s generate-embeddings phase applies here with a sharper edge: the query must use the same embedding model and the same output width as the chunks.

Truncation mismatches are a common way to get there. Indexing with OpenAI text-embedding-3-large at 3072 dimensions and querying with dimensions set to 256 without re-embedding the corpus mixes two contracts. Indexing a Matryoshka prefix while a batch job still writes full-width vectors into the same collection does the same. Collections in Weaviate, Pinecone, Qdrant and Milvus are created with a fixed vector dimension; changing that dimension is a new collection or a migration, not a config toggle.

Some stacks fail closed: the vector database returns an error when the query vector length disagrees with the collection schema. That is the better failure. Others fail open if application code pads with zeros or truncates without logging. Treat any pad-or-slice shim in application code as a defect. The store’s dimension field is the contract; application code should not invent a second contract to paper over a deploy mistake.

Operational detection looks like sudden recall collapse after a client library change, a new embed worker, or a “temporary” shortening flag. Log the embedding model id and the output dimension with every write and every query. When those fields disagree, stop and re-embed rather than A/B testing prompts. Choosing the right width up front reduces how often you hit that incident.

How do you choose embedding dimensions for RAG?

You choose embedding dimensions for RAG by fixing the embedding model family first, then picking the smallest width that holds your labelled recall under your storage and latency budget. Model selection owns language, domain and hosting; that procedure lives on how to choose an embedding model. Once the family is fixed, Particula’s rule is start at 384 to 768 for well-structured business content, measure precision or recall at the top-k your prompt reads, and only move to 1024, 1536 or 3072 when the metric gap is real.

Use the provider’s native shortening when the model is Matryoshka-aware, so you can A/B 256, 512, 1024 and full width without training a new network. Re-run a truncation test in the spirit of Milvus’s STS-B method on your own pairs before you cut a large collection to 256 dimensions. If storage still dominates after a sensible width, add quantization as studied in arXiv 2505.00105 rather than assuming another doubling of dimensions will fix recall. Domain vocabulary failures after a fair width test belong on fine-tuning embeddings for your domain, not on a blind jump to 3072.

When two widths tie on your labelled set, prefer the smaller width and keep the larger as a documented upgrade path, the same iterate-from-baseline habit Weaviate recommends for model choice. Pair the width decision with an ANN parameter review: HNSW ef and M values that were tuned at 384 dimensions may need a fresh latency-versus-recall pass at 1536 dimensions because the distance computations got more expensive per candidate. Store choice among Weaviate, Pinecone, Qdrant and Milvus does not remove that tuning step.

Document the chosen width beside the model id in the index metadata. The next engineer should not have to guess whether vectors are full-width or truncated prefixes. Implementation is then a small set of API and normalisation steps rather than a research project.

How do you implement dimension truncation?

You implement dimension truncation by requesting the shortened vector from the provider when possible, normalising if you slice locally, creating the vector collection at that exact dimension, embedding both documents and queries to that width, and evaluating before cutover. For OpenAI text-embedding-3 models, pass the dimensions argument on the embeddings create call so the API returns the prefix directly (OpenAI, 2024). For Sentence-Transformers Matryoshka models, truncate to the trained widths and normalise. For stores, set the collection dimension to the truncated size in Weaviate, Pinecone, Qdrant or Milvus so inserts that disagree on length fail loudly.

A minimal Python pattern with OpenAI’s API and a Weaviate collection configured for 256-dimension vectors looks like this when you already have the Weaviate client connected and the collection schema fixed:

from openai import OpenAI
import weaviate
from weaviate.classes.config import Configure, Property, DataType

DIM = 256
oai = OpenAI()
client = weaviate.connect_to_local()

client.collections.create(
    "Chunk",
    properties=[Property(name="text", data_type=DataType.TEXT)],
    vector_config=Configure.Vectors.self_provided(),
)

def embed(text: str) -> list[float]:
    v = oai.embeddings.create(
        model="text-embedding-3-large",
        input=text,
        dimensions=DIM,
    ).data[0].embedding
    return v  # already length DIM from the API

chunks = client.collections.use("Chunk")
vec = embed("gateway returned error code 502")
chunks.data.insert({"text": "gateway returned error code 502"}, vector=vec)

Batch backfills should checkpoint progress by document id and write the target dimension into every object’s metadata so a crashed job cannot resume with a different width. Run a shape assertion in continuous integration: embed a fixture string and fail the build if len(vector) != DIM. After cutover, watch p95 query latency and recall on a canary slice of traffic for at least one full business day before deleting the full-width collection.

Pin library versions in your lockfile (openai and weaviate-client as of September 2026) and date any unstable embedding API fields. After backfill, compare recall at k against the full-width baseline on the same labelled queries. If the drop is unacceptable, raise dimensions or keep full width and compress with quantization instead. Truncation that passes offline eval can still drift later if a worker forgets the dimensions argument, which is why logging model id and width on every embed remains part of the implementation, not an optional extra.

IBM’s RAG cookbook frames the same performance-versus-cost trade-off when it discusses embeddings on watsonx: larger representations and larger models can improve quality while raising the bill, so teams should treat dimension and model size as paired dials rather than independent upgrades. Azure’s compare-and-evaluate guidance similarly tells you to experiment with dimensionality settings and judge them with retrieval metrics on your use case, not with a generic preference for maximum resolution.

Training your own Matryoshka model is optional. Most teams will never open MatryoshkaLoss. They will pick a hosted or open model that already documents truncation behaviour, verify the quality curve on their labelled pairs, and fix the collection dimension to the winning prefix. Custom MRL training belongs with teams that already fine-tune embedders for domain vocabulary and want nested widths as part of that training run.

Particula’s million-document storage example is worth repeating in planning docs: about 12 GB of vector storage at 3072 dimensions versus about 1.5 GB at 384 dimensions for the same count, before ANN structures. That gap shows up as RAM for in-memory indexes, as disk for on-disk segments, and as network bytes when replicas rebuild. Teams that treat dimensions as a free accuracy upgrade discover the bill when the third replica comes online.

If you need still more compression after a fair width, read quantization results as attributed experiments: arXiv 2505.00105 and Shakir et al. (2024) show large byte savings with careful re-scoring, not a free lunch. Width first, precision second, and always the same contract at write and query time. GMI Cloud, Beam, GreenNode and Galileo all list dimensionality beside latency when they describe production RAG infrastructure, which is the right company for this decision to keep. Write the chosen width into the runbook next to the embedding model id so capacity planning, cost forecasts and incident response share one number across the whole retrieval team.

Keep a one-page decision record: model id, output dimensions, truncation method, labelled-set size, recall at the prompt top-k, storage estimate at float32, and the date of the bake-off. That record is what makes the next re-embed a deliberate migration instead of a guess.

Embedding dimensions are a capacity dial and a cost dial at the same time. Particula’s production evidence favours 384 to 768 for many business corpora, with higher widths reserved for measurable hard cases. Matryoshka Representation Learning (Kusupati et al., 2022) and provider dimensions APIs make shorter prefixes usable; Milvus’s truncation bake-off shows that not every model survives a cut to 256 dimensions equally. Storage scales roughly with dimensions times bytes per value, as the one-million-vector float32 table and arXiv 2505.00105 both illustrate. Choose the model first, pick the smallest width your labelled recall can afford, truncate safely with matching index and query contracts, and re-embed when the width changes. Mixing dimensions is how an index goes quiet and wrong.