The Infrastructure RAG Runs On

RAG infrastructure is the storage, indexing, embedding and orchestration layer around retrieval, and it decides what the system can find, how safely it can return it, and how quickly that context reaches the model. Google Cloud Architecture Center’s reference for RAG with Gemini Enterprise and Agent Platform (last reviewed 10 November 2025) splits that work into a data ingestion subsystem and a serving subsystem: embeddings are built into a searchable vector index on the way in, then a query is embedded, searched, augmented and answered on the way out. AWS’s RAG overview describes the same loop as create external data, retrieve relevant information, augment the LLM prompt, and update external data. Introl’s December 2025 production guide treats vector databases, embedding pipelines, chunking and retrieval orchestration as distinct infrastructure challenges that compound at scale. On this site the layer matters only through what determines whether the answer can be right: the practical next choices are which vector store, which embedding model, which framework defaults, and whether relationships justify a graph substrate.

What is RAG infrastructure?

RAG infrastructure is the layer that stores chunks, turns them into embeddings, indexes them, filters them, and hands the retrieved context back to the model. Canonical’s Ubuntu blog on RAG AI infrastructure (part of its open-source Gen AI series) frames the problem as operating the hardware, cloud, operating system and Gen AI services that keep retrieval and generation reliable, while AWS and Google Cloud’s product pages frame the same idea as connecting a large language model to proprietary or fresh external data. Wikipedia’s Retrieval-augmented generation entry (process section) names the familiar stages: retrieve, then generate from an augmented prompt. Those definitions agree on the job and disagree on how much “infrastructure” should include hardware catalogues versus retrieval substrates.

The useful border for a retrieval engineer is narrower. Infrastructure here means the adjacent stores and services that change what retrieval can find, how it can filter, how tenants stay isolated, and how context is assembled for generation. Chunk size, BM25 scoring, cross-encoder reranking and prompt assembly still decide answer quality, but they are pipeline decisions owned by the core stage hubs rather than author-layer foundations. Jam with AI’s “Mother of AI” infrastructure write-up (Substack) pushes teams to build a production foundation before polishing demos, which is the same border stated from the builder side: without a durable ingest and index path, serving tricks do not compound.

That author-layer rule is deliberate. A page that catalogues every GPU SKU, every Kubernetes operator and every managed agent console will look complete and still leave the retrieval engineer without a decision order. The four hubs under this parent exist so each decision can carry depth without forcing every reader through a cloud-provider reference architecture.

Once that border is fixed, the next question is how the data path actually splits between building the index and answering a live query. Google Cloud’s architecture document and AWS’s four-step loop give the same answer with different product names, and that split is the spine for every later choice.

How does RAG infrastructure split into ingestion and serving?

RAG infrastructure splits into an ingestion path that prepares searchable data and a serving path that answers queries against that index. Google Cloud Architecture Center (10 November 2025) makes the split explicit: the ingestion subsystem uploads sources, enriches metadata, parses and chunks documents, and builds vector embeddings into a managed datastore; the serving subsystem takes a user query through a frontend and backend, embeds the query, runs semantic search, augments the prompt, and returns a grounded response. AWS’s four-step description is the same loop in product language: create external data, retrieve, augment, and keep the external store current.

Introl’s production guide expands the ingestion side into document pipelines, chunking systems and embedding infrastructure, and expands the serving side into vector search plus orchestration that may add caching, fallbacks and observability. Render’s build-versus-buy article (2025 framing) treats the same split as an operations problem: fragmented stacks pay an integration tax on every hop between ingest jobs, vector storage and application hosting. Google Cloud’s enterprise requirements call out real-time availability after a source lands in object storage, and enriched metadata so search can narrow by product line, author, location or document type before similarity ranks the rest.

The architectural lesson is stable across those sources even when the vendor products differ: if freshness, permissions or embedding versioning break on the ingest side, serving cannot recover them with a better prompt. That is why what each stage does in the pipeline still sits upstream of every infrastructure choice. An index that never received a delete, a chunker that split a table mid-row, or an embedding model swap that left old vectors in place will show up as wrong answers at serving time. With the dataflow clear, the useful cut is which infrastructure decisions actually move retrieval quality rather than only renaming cloud services.

Which infrastructure decisions change retrieval quality?

Four infrastructure decisions change retrieval quality more than the rest of the catalogue: which vector store holds the embeddings, which embedding model defines the space, which framework decides orchestration defaults, and whether a knowledge graph belongs beside similarity search. Introl’s December 2025 guide spends separate sections on vector database selection, embedding model selection and orchestration frameworks, then treats GraphRAG as an advanced pattern when relationships matter. Ubuntu’s OPEA-oriented reference table lists vector databases, embedding models, retrieval and ranking, prompt processing and frameworks as distinct services. Galileo’s architecture guide walks multi-stage pipelines, chunking, GraphRAG and agentic loops as pattern choices on top of the same spine.

Those lists sprawl into chunking recipes, hardware SKUs and full agent platforms. The retrieval-relevant collapse is the four-way map below. Storage and indexing live under what they change about retrieval. Representation lives under which properties predict retrieval performance. Orchestration abstractions live under what they decide for you. Relationship substrates live under what a graph makes retrievable. Cloud hardware catalogues, operating-system comparisons and bare-metal versus public-cloud matrices (Ubuntu’s opening sections) sit outside this author layer because they do not change nearest-neighbour behaviour by themselves.

Chunking strategy, hybrid fusion maths and reranker selection still dominate quality, which is why Introl and Galileo devote long sections to them. Those topics are owned here by chunking, retrieval and reranking, not by repeating their mechanism depth inside an infrastructure overview. With the four decisions named, the storage decision is the one most explainers treat as the entire subject, so it opens next.

How do vector databases change what retrieval can return?

Vector databases change what retrieval can return by controlling index type, metadata filtering, hybrid lexical-plus-vector scoring, and tenant isolation around the embedding store. Introl’s December 2025 guide lists Weaviate, Pinecone, Milvus and Qdrant as the four options it treats as market defaults, each with a different operations profile: hybrid and modules on Weaviate, managed zero-ops on Pinecone, distributed scale on Milvus, and filtering-first design on Qdrant. Weaviate leads the ordered field on this site, followed by Pinecone, Qdrant, Milvus and the wider set. Ubuntu’s open-source Gen AI service table also lists relational and lexical engines that grew vector modes (Postgres with a vector extension; OpenSearch k-NN), which is useful as a map of shapes rather than as a closed shortlist. Google Cloud’s RAG use-case page pairs vector databases with relevancy re-rankers when it explains how search finds context for generation.

Not every option in that set is the same kind of product. Weaviate, Pinecone, Qdrant and Milvus are purpose-built vector databases or managed vector services. Weaviate is still compared with library and extension options that are not full databases: FAISS is a similarity-search library, and pgvector is a Postgres extension, and both still appear in RAG decisions when searchers say “vector database”. Weaviate is also compared with search and cache engines that added vector fields later: Elasticsearch and OpenSearch keep a lexical core, and Redis keeps an in-memory core, each with a different operations profile. The category overview, the property checklist and the profile router live at vector databases for RAG. The scored fit across the common choices, with method stated beside the table, lives at which vector database should you use for RAG.

Index internals such as HNSW parameters, IVF partitions and quantization are retrieval mechanics that the storage product exposes. Those mechanics are owned by the indexing cluster once you already know you need a vector layer. The storage decision is settled only when filtering, tenancy and operations match the workload; the next infrastructure decision is what those vectors actually encode.

How do embedding models and frameworks shape the RAG stack?

Embedding models shape the RAG stack by defining the vector space the index searches, and frameworks shape it by choosing defaults for loaders, retrievers, prompts and retries. Introl’s embedding section (December 2025) compares managed APIs such as Voyage AI, OpenAI and Cohere with self-hosted open-source families such as BGE, E5 and GTE, and stresses that changing models later forces a full re-embed. Ubuntu’s service table places sentence transformers and framework choices beside the vector store as peer services. Galileo’s pipeline writing treats embedding and orchestration as stages you can swap without renaming the whole architecture.

The retrieval consequence is blunt. A weak embedding space will not be fixed by a faster ANN index, and a framework default that silently drops metadata filters will look like a vector-database bug. Model properties that predict retrieval behaviour (dimensions, multilingual coverage, domain fit) are covered under embedding models for retrieval, with the selection decision at the embeddings cluster. What LangChain, LlamaIndex, Haystack and neighbouring tools decide for you, and what those defaults hide, is covered under RAG frameworks. Neither hub replaces measuring recall on your own queries; both stop the stack from being chosen by tutorial inertia alone.

Introl also warns that embedding cost and storage scale with token volume and dimension, and that quantization is one way teams cut storage once the model is fixed. Those numbers move by vendor price list and by corpus shape, so they are not restated here as fixed site figures. The operational point stands without a fake price: the embedding choice locks a migration cost, and the framework choice locks which retrieval knobs you can see. Once vectors and orchestration are chosen, some questions still fail because similarity is the wrong retrieval primitive. That is the graph question, and it is the fourth infrastructure decision rather than a marketing add-on.

When should RAG infrastructure include a knowledge graph?

RAG infrastructure should include a knowledge graph when the question depends on relationships, entity traversal or explainable multi-hop paths that pure similarity search does not preserve. Introl’s advanced-architecture section describes GraphRAG as building entity graphs at ingest and traversing edges at query time, citing Microsoft’s GraphRAG line of work for narrative private data. Galileo’s architecture guide pairs GraphRAG with agentic and self-corrective patterns as alternatives to a single vector hop. Wikipedia’s improvements discussion treats hybrid search and graph-oriented retrieval as part of the broader RAG research line rather than as a mandatory default.

Vector search remains the default foundation for unstructured corpora. A graph earns its keep when “who reports to whom”, “which clause depends on which schedule”, or “which lab result links to which protocol” is the actual question. Google Cloud’s legal-research and clinical-assistance examples in the same architecture family show prompts that need proprietary corpora plus domain structure, which is where teams start asking whether similarity alone is enough. The substrate trade-off, including construction cost, lives at knowledge graphs as a retrieval substrate. The architecture pattern built on that substrate lives at GraphRAG.

Choosing a graph without fixing chunking, permissions and evaluation usually multiplies ops cost without fixing the failure mode you already had in the vector path. Introl’s own precision claims for GraphRAG are Introl’s reporting of Microsoft-linked work and are not restated here as an independent score. The practical test is whether your failure cases are missing edges or missing neighbours: missing edges push toward a graph substrate; missing neighbours push toward better embeddings, chunking, hybrid search or reranking first.

With all four decisions named, production teams still hit a different class of infrastructure pain: whether to operate the stack themselves, and which operational bands kill a demo after it looked fine on a laptop.

Do you need a vector database for RAG?

Most semantic RAG systems need a vector index or a vector-capable search layer, because the serving path embeds the query and must find nearest neighbours in a corpus that no longer fits in the prompt. AWS, Google Cloud and Ubuntu all describe that pattern when they explain how retrieved passages ground generation. Meilisearch’s contrarian essay against dedicated vector databases (present on related vector-database queries) argues that lexical search plus good ranking can cover many RAG jobs; that argument is a useful stress test, not a universal veto. The decision is whether your queries need semantic neighbourhoods, exact lexical hits, structured filters, or a hybrid of those modes.

A dedicated vector database is the right default when embedding search is central, filters and tenancy must travel with the vectors, and you want an API shaped around ANN retrieval. A library such as FAISS can be enough inside a custom service when you already operate storage, replication and metadata elsewhere. A relational extension such as pgvector can be enough when the corpus is modest and transactional data already lives in Postgres. An existing search engine with vector fields can be enough when BM25 corpora are already the system of record. The category map for those shapes is vector databases; the scored product choice is the scored vector-database survey.

Skipping a vector layer entirely only works when the retrieval problem is truly lexical or structured, or when the entire working set fits a long-context strategy you have measured. Those exceptions are comparison topics under the decisions cluster, not a reason to leave the storage decision unstated. With the “need” question answered at category level, build-versus-buy and production operations decide who runs the layer you picked.

How do build-versus-buy and operations show up in RAG infrastructure?

Build-versus-buy and operations show up in RAG infrastructure as choices about who runs the vector layer, who pays the embedding bill, and which production bands (freshness, scale, tenancy, latency, cost) the demo never exercised. Render’s build-versus-buy article frames fragmented DIY stacks as an integration tax and argues for unified platforms when teams lack persistent ingest compute, while still naming workloads that need custom sharding or air-gapped networks. Introl’s decision section recommends starting with managed vector and embedding services, then graduating to hybrid self-hosting as volume and control requirements rise. Google Cloud Architecture Center’s design alternatives compare Cloud Run, Agent Runtime, GKE and Compute Engine on configuration flexibility versus management effort for the application tier around the same RAG dataflow.

Introl’s scaling chapter (December 2025) lists embedding throughput, vector index memory pressure, query latency targets, embedding drift after model changes, multi-tenant isolation and cost growth as the bands that appear after a prototype. Those bands are real; the universal millisecond and dollar figures Introl quotes are Introl’s own reporting for its narrative, not independent benchmarks published here. Galileo’s production section adds evaluation and hallucination monitoring as the measurement layer that tells you whether the infrastructure is still serving the right context. The owning depth pages are RAG cost, why a RAG pipeline is slow, retrieval quality degradation, incremental indexing, multi-tenancy, how to evaluate a RAG system and tracing.

Do you need a vector database for RAG? Most semantic RAG systems need a vector index or a vector-capable search layer, but the right product depends on filtering, tenancy, hybrid search and operations rather than on a vendor logo. When should RAG use a knowledge graph? When relationship traversal beats similarity alone. How is RAG implemented at the infrastructure layer? As ingestion plus serving, then the four decisions above. The scored storage choice, when you are ready to pick a product rather than a category, is scored storage choice for RAG.

RAG infrastructure is the adjacent layer that turns documents into a searchable substrate and keeps that substrate answerable under live traffic. Google Cloud Architecture Center’s ingestion-versus-serving split, AWS’s create-retrieve-augment-update loop, and Introl’s component list all describe the same job with different product names. The retrieval-relevant decisions are four: vector store, embedding model, framework defaults, and whether a graph belongs beside similarity search. Route each decision to its hub, keep chunking and scoring in the pipeline hubs that own them, and treat managed-versus-self-hosted ops as a constraint on those decisions rather than as a substitute for measuring what the index actually returns.