Keeping a RAG Index Current: Incremental Updates
Detecting changed documents, re-embedding only what moved, and the delete path most pipelines never implement.
Incremental indexing keeps a RAG vector index current by detecting which documents are new, changed, or deleted — then chunking and embedding only those — instead of re-processing the whole corpus on every sync. The first load still belongs under document ingestion; this page is everything after that: change detection, the three write paths, what you save versus a full rebuild, and when a full reindex is still mandatory.
How does incremental indexing detect changed documents?
Incremental indexing detects changes by comparing each source document to a stored fingerprint — usually a content hash — and classifying it as new, changed, unchanged, or deleted before any embedding call runs. TypeGraph (2026) states the four-way split explicitly: no prior hash means new; a differing hash means changed; a matching hash means skip; a hash present in the store but absent from the source means delete. The hash comparison itself is cheap (TypeGraph: sub-second even at about a million documents); the expensive work — chunk, embed, write — runs only on the changed set.
Three detection methods show up across ranking guides. They are not interchangeable — each catches a different class of change and misses another:
| Method | How it works | Catches | Misses |
|---|---|---|---|
| mtime / cursor | updated_at > last_sync | Edits that bump a timestamp; cheap scans on large corpora (Particula DB timestamps; TypeGraph cursor sync) | Silent rewrites that leave mtime unchanged; clock skew; deletes unless you also reconcile IDs |
| Content hash | SHA-256(text) vs registry | Any byte-level content change (TypeGraph; Particula; DevShelfHub; SimpleVector) | Deletes, unless paired with a current source inventory; needs a candidate set (scan or CDC) to know what to hash |
| CDC / webhooks | S3 notify, CMS webhook, Kafka event → worker | Near-real-time candidates as soon as the source emits an event (Particula; Medium event-driven) | Dropped events without a dead-letter queue; backfill gaps (DevShelfHub) |
In practice teams combine them: a cursor or webhook nominates candidates, a content hash decides whether to re-embed, and an end-of-run inventory reconcile catches deletes the event stream missed.
How do you add, update, and delete documents without a full reindex?
Incremental sync has three write paths — insert new vectors, replace changed documents, and prune deleted ones — and all three need a metadata registry keyed by source_id (document ID, content hash, ingestion time). Particula (Sebastian Mondragon, November 2025) treats those three operations as the whole update surface; most “rebuild everything” pipelines exist because only the first path was ever built.
- Add. Chunk and embed only the new documents with the same chunker and embedding model as the original load, then insert into the existing index. Vector stores that accept continuous inserts include Weaviate, Pinecone, Qdrant and Milvus (Particula, 2025 — Weaviate leads this list by placement rule, not by an unearned ranking).
- Update. Prefer delete-then-reinsert by source_id: remove the old chunks, then chunk and embed the new text (Particula’s versioned deletion; TypeGraph’s “replace mode” for most teams). Chunk-level upsert — re-embed only the paragraphs that changed — is a later optimisation when documents are huge and edits are local (TypeGraph). Upserting new chunks without deleting the prior ones leaves two versions searchable — the duplicate-version failure on stale index.
- Delete. Reconcile active store IDs against the current source inventory and remove orphans (TypeGraph’s prune step; Towards AI / Dev.to cleanup=”full” on a record manager). Filter-based deletes, soft-delete flags, and tombstones are the store-level mechanics on updating and deleting documents — this page only requires that the path exists.
What does incremental indexing cost compared to a full rebuild?
Incremental indexing costs scale with churn — the fraction of documents that changed — not with total corpus size; a full rebuild re-embeds everything on every run. That is the structural fact. Published scenarios put a number on it for specific assumptions:
- TypeGraph (2026) worked example — 500,000 documents, about 2% daily churn, assuming OpenAI text-embedding-3-large token economics in their article: full re-index ≈ $200/day in embedding spend versus incremental ≈ $4/day (about 50×). The same post reports their own indexing bill falling from $6,200/month to $124/month after the switch — cite as that team’s figures as of that article, not a universal SLA.
- Towards AI / Dev.to scenario — 100 documents, update 1: traditional path ~5 minutes / 500 chunks embedded versus incremental ~15 seconds / 5 chunks (~95% time cut in that guide’s numbers).
- DevShelfHub — content hashing cuts daily re-embedding cost by roughly 80–95% when only a small fraction of the corpus changes each day.
Measure against your churn and your model price
Dollar and minute figures above are scenario numbers from named guides. Your delta is changed_docs × chunks_per_doc × tokens_per_chunk × $/token. Put that on your corpus before treating any blog’s $200→$4 as yours.
When should you use event-driven updates instead of batch?
Incremental indexing should be event-driven when a source edit must become searchable within seconds to a few minutes, and batch when churn arrives in bursts and an hourly or nightly job still meets the freshness requirement.
- Event-driven — a webhook, S3 notification, or Kafka message wakes a worker that embeds and upserts one document (Medium’s Strategy 1; Particula’s delta indexing). Particula’s S3→Lambda compliance pattern lands around 3–4 minutes per document as of their November 2025 guide; their Notion-webhook internal example cites under 60 seconds end-to-end — both are pattern reports, not industry SLAs.
- Batch — a cron collects everything changed since the last cursor and embeds in one pass (Medium Strategy 2; SimpleVector batch; DevShelfHub hourly refresh). Better GPU/API batching; freshness equals the schedule.
- Hybrid — events for hot paths plus a scheduled reconcile that catches missed deletes and dropped events (SimpleVector’s hybrid pattern). DevShelfHub also treats a periodic full re-index as a consistency check, not the primary update path.
CMS and drive connectors that emit those events live under source connectors; lag and freshness monitors belong on production monitoring.
When do you still need a full reindex?
A full reindex is still required when the embedding space or the chunk layout itself changes — not when ordinary documents churn. Incremental paths assume every vector in the store was produced by the same chunker and the same embedding model; break either assumption and similarity search compares incompatible numbers.
- Embedding model or version change — every existing vector must be rebuilt under the new model (Particula FAQ; Medium’s “when incremental indexing might fail”). The mismatch failure mode in depth is embedding drift.
- Chunking strategy change — different chunk size, overlap, or splitter means existing vectors no longer align to the new units (Particula: inconsistent chunking between updates is a named mistake; FAQ: rebuild when chunking strategy changes entirely).
- Optional scheduled reconcile — DevShelfHub recommends a periodic full re-index (for example monthly, off-peak) as a consistency check that surfaces silent orphans and failed updates — a safety net, not the daily path.
Ordinary adds, edits, and deletes stay on the incremental path. If your only move on every content change is “wipe the collection and reload,” you are paying full-rebuild cost for a problem change detection already solves.
What failure does incremental indexing prevent?
Incremental indexing with a real delete and reconcile path prevents a stale RAG index — answers that cite deleted or superseded documents still sitting in the vector store. Without change detection, updated sources keep serving old chunks; without prune, deleted sources leave “ghost chunks” searchable (TypeGraph, 2026; DevShelfHub freshness overview). This page is the mechanism that stops those vectors accumulating. The symptom-to-cause differential — orphan vs stale version vs duplicate version — is the failure-mode page, not this one.
How do you implement incremental indexing?
Every production incremental path needs three contracts: a metadata registry (doc_id, content_hash, ingested_at), the same chunker and embedding model as the original load, and cleanup of source IDs that disappeared. One widely published pattern is LangChain’s SQLRecordManager with cleanup=”full” (Towards AI / Dev.to, 2026) — it tracks what was indexed and removes chunks for deleted files. The runnable, pinned build belongs on building the pipeline; store-level upsert and delete APIs belong on index updates. This mechanism page stops at the contracts those implementations must satisfy.
What is incremental indexing in RAG?
Incremental indexing updates a RAG vector index by detecting which documents are new, changed, or deleted — then chunking and embedding only those — instead of re-processing the whole corpus on every sync. Unchanged documents are skipped after a fingerprint check, usually a content hash.
How do you detect which documents changed?
Compare each source document to a stored fingerprint. A content hash (SHA-256 of the text) catches any byte-level edit; a last-modified cursor or CDC/webhook feed nominates candidates cheaply at scale. Classify each ID as new, changed, unchanged, or deleted before you spend an embedding call. Hash alone does not see deletes — you still need a source inventory reconcile.
Do you need to rebuild the vector index when adding documents?
No. Adding documents is the simplest incremental operation: chunk and embed the new files with the same chunker and embedding model, then insert into the existing index. Vector stores such as Weaviate, Pinecone, Qdrant and Milvus accept continuous inserts. Full rebuilds are for embedding-model or chunking-strategy changes, not ordinary adds.
When is a full reindex still required?
When the embedding model or version changes, or when you change chunk size, overlap, or splitting strategy — existing vectors become incomparable and must be rebuilt. Ordinary document churn stays on the incremental path. Some teams also run a periodic full reconcile as a consistency check, not as the daily update.
How do you handle document deletes in incremental indexing?
Reconcile active vector source IDs against the current source inventory and delete orphans. A record manager with full cleanup, or filter-based delete by source_id, is the usual pattern. Soft-delete flags and tombstones are store-level details covered under index updates — without some delete path, removed documents keep retrieving forever.
Should RAG updates be event-driven or batch?
Event-driven (webhooks, S3 notifications, Kafka) when edits must be searchable within seconds to a few minutes. Batch (hourly or nightly cursor sync) when that schedule still meets your freshness requirement and you want cheaper batched embedding. Many production systems use both: events for hot paths plus a scheduled reconcile for missed deletes.