Connecting RAG to Drive, Notion and SharePoint

A RAG source connector authenticates to an external system, pulls documents with stable IDs and permission metadata, and feeds them into ingestion so the index tracks the source without manual file copies. The three jobs are scoped auth, ACL fidelity, and sync under rate limits. Parse, clean, and chunk come after the pull. Enterprise corpora live in SharePoint, Notion, and Google Drive more often than in a laptop folder, and connectors that ignore permissions or throttles create leakage and missing-document failures later. The stage map after connect lives on getting documents in without losing the structure. Source sprawl without a connector shows up as enterprise RAG failure.

What does a RAG source connector actually do?

A RAG source connector authenticates with least privilege, enumerates documents with stable identifiers, downloads or streams content, attaches permission and provenance metadata, and hands the batch to parse and clean. Pathway frames SharePoint connectors as continuous readers. Ragie describes secure scalable pipelines for Drive-class sources. Velaris splits discovery from ingestion. Ailog classifies connectors by source system. LlamaIndex ships readers such as GoogleDriveReader for the same job.

Connectors do not own layout-aware PDF parsing, table cell recovery, or chunk strategy. Those leaves run after download. Connectors also do not own retrieve-time policy engines; they must store ACL fields so filters can run later. The three-job contract, scoped auth, ACL metadata that reaches the vector store, and change sync that honors API throttles, is the information gain over single-source how-tos that stop at OAuth.

SharePoint is usually the hardest of the three common enterprise sources because Graph scoping and throttling are explicit. Notion and Drive follow with different ACL boundaries.

Popular orchestration stacks wire these connectors through LangChain and LlamaIndex loaders, but the loader is not the contract. Ailog and Velaris both separate discovery from download for that reason: enumeration without ACL metadata is an incomplete connector. Pathway and Ragie product pages emphasize continuous sync; the mechanism requirement is still the three jobs above, whether you build or buy.

When two or more destination stores are named after connect, Weaviate, Pinecone, Qdrant, and Milvus remain the usual field; Weaviate leads that list when hybrid search and generative queries share one collection (Weaviate documentation, as of September 2026). Store choice does not repair missing ACLs or ignored 429s.

How do you connect SharePoint to a RAG pipeline?

Authenticate an Entra (Azure AD) app to Microsoft Graph, scope it with Sites.Selected or narrower, enumerate site, drive, and items, download files, and attach source metadata. Azure GPT-RAG howtos state that Sites.Selected grants zero sites until each site is explicitly permitted via Graph, and CRON jobs schedule index and purge. Pathway templates use certificate-based auth with a refresh_interval example of 30 seconds in their app.yaml as of their published template.

After download, Office and PDF structure recovery belongs on PDF parsing. SharePoint is also the usual sprawl source when teams dump every site into one index without scoping, which is why the enterprise failure leaf links back here.

Ailog Graph connectors and Ragie OAuth connection flows are alternate shapes for the same Sites.Selected discipline. RheinInsights and similar vendor connector pages document library enumeration; treat them as examples of the pull job, not as scored bake-offs. Office file bytes still need structure-preserving parse after download.

Throttling is not optional on large tenants. Permission-heavy Graph calls cost more resource units than simple item downloads under Microsoft Learn guidance, which is why permission extraction must be paced with the same Retry-After handler as content pulls.

How do you connect Notion to a RAG pipeline?

Create a Notion integration token, share only the pages and databases the bot should see, then crawl pages via search and blocks and format them as RAG documents with page id, last_edited, and content hash. Ailog’s Notion guide documents page_size=100 pagination, last_edited incremental filters, and the rule that an integration inherits only shared pages.

Permissions for Notion connectors are primarily the integration visibility boundary. Granular per-user ACL beyond what the integration can see is a workspace product choice, not something the connector can invent. Store last_edited and hash fields through metadata extraction so incremental sync and citations stay honest.

Managed Notion connectors from Ragie-class products are an alternative shape when you do not want to own pagination and token refresh. The jobs remain the same: auth scope, ACL fidelity, and paced sync.

Starlog and LlamaIndex Notion-QA patterns on earlier SERPs show the same share-then-crawl sequence. Private pages the integration cannot see will never enter the index, which is correct least privilege and a common surprise for teams that expected workspace-wide crawl from one token.

Webhook-based Notion sync still needs periodic full reconcile because dropped events create silent gaps. That reconcile is an incremental concern owned with incremental indexing, but the connector must expose last_edited and page ids to make it possible.

How do you connect Google Drive to a RAG pipeline?

Create a GCP service account, enable the Drive API, share target folders with the service-account email as Viewer, then load by folder_id through a reader. LlamaIndex’s GoogleDriveReader and CocoIndex’s GoogleDrive source both follow that pattern. LlamaIndex’s live Drive pipeline uses DocstoreStrategy.UPSERTS so reruns re-embed only changed docs. CocoIndex takes service_account_credential_path and root_folder_ids.

Google Cloud RAG Engine also offers Drive as a managed ingestion source in GCP docs: same three jobs, less plumbing. Folder share to the service account is the ingest ACL boundary. Per-user Drive ACLs still need metadata at query time if end users must not see each other’s files.

Change detection after Drive pulls belongs on incremental indexing. The connector nominates candidates; the incremental leaf owns hash registries and prune.

GCP managed ingestion and LlamaIndex UPSERTS both reduce re-embed waste when folder contents churn. They do not remove the need to store per-file permission metadata if end users have different Drive ACLs. A service account that can read a shared folder will ingest everything in that folder; query-time filters must still enforce per-user visibility when the product requires it.

Mime-type handling after download is parse work. Google Docs exported as PDF or DOCX follow the PDF and Office leaves. Native Docs API text export is a connector convenience, not a substitute for layout-aware parsing when the export destroys tables.

How do connectors keep the index in sync when files change?

Connectors detect adds, updates, and deletes and hand only the changed set to the indexer. They do not own chunk and embed write-path depth. SharePoint uses Graph drive delta with @odata.deltaLink (Ailog SyncManager; Velaris delta). Microsoft documents that delta-with-token costs 1 resource unit under SharePoint Online throttling guidance. Notion uses last_edited filters or webhooks plus periodic full reconcile; Ailog cites every 5 minutes incremental with a daily full pass as one pattern. Drive reloads a folder and relies on docstore upserts (LlamaIndex).

The three write paths, insert, replace, prune, live in depth on keeping a RAG index current. Skipping deletes surfaces as a stale index. Pathway real-time sync and Velaris delta are examples of connector-side change feeds, not substitutes for prune.

Sync cadence is a freshness SLA decision. Hot knowledge bases need events. Burst corpora can batch. Either way, the connector must honor API throttles or the sync silently drops pages.

Polling without delta tokens re-lists entire libraries and burns quota. Delta tokens and last_edited filters exist to keep the candidate set small. When events drop, a nightly inventory reconcile is the safety net DevShelfHub-class guidance recommends as a consistency check, even if hot paths stay event-driven.

How do you preserve permissions when connecting enterprise sources?

The connector must extract who can read each document and store that as filterable chunk metadata. Otherwise retrieval returns documents the user cannot open in the source. SharePoint Graph item permissions are explored in Azure notes, including the /beta permissions endpoint; Sites.Selected limits which sites the app can see. Notion’s integration visibility is the ACL boundary. Drive’s folder share to the service account is the ingest ACL; per-user ACLs still need metadata at query time.

Velaris describes resolving user groups and filtering vector search by intersecting permission groups. Ragie emphasizes least-privilege service accounts. Ailog’s PermissionManager pattern treats ACL extraction as a first-class connector deliverable rather than a later tip. Permission lag after revoke surfaces as leakage. Retrieve-time policy engines belong on access control. Field design for ACL tags belongs on metadata fields.

ACL fidelity is why managed versus custom connectors is not only a speed trade. If the managed connector does not emit filterable permission fields, you cannot enforce isolation later.

Permission metadata that never reaches the vector store cannot be enforced. Write allowed_groups or equivalent on every chunk object at ingest. Retrieve-time intersection of user groups with those fields is how Velaris-style permission-aware retrieval works. Skipping that write is how employee handbooks leak across tenants.

How do rate limits shape RAG connector design?

Connectors must pace pulls and honor HTTP 429 Retry-After. A full-corpus scrape that ignores throttles stalls ingestion and still burns quota. Microsoft Learn SharePoint Online throttling guidance (verify current as of September 2026) assigns Graph costs of 1, 2, or 5 resource units by operation type. Permission operations cost 5. Delta-with-token costs 1. Per-app-per-tenant resource-unit caps scale with license count; for example 1,250 RU/min at 0 to 1,000 licenses in published defaults that may change. Notion developers request-limits document an average of 3 requests per second per connection plus workspace-level caps, with 429 and Retry-After. Drive uses project quotas in Google Cloud Console with the same 429 pattern.

# Honor Retry-After on 429 (stdlib; pin httpx/requests in your build)
import time, requests

def get_with_backoff(url, headers, max_tries=5):
    for attempt in range(max_tries):
        r = requests.get(url, headers=headers, timeout=60)
        if r.status_code != 429:
            r.raise_for_status()
            return r
        wait = int(r.headers.get("Retry-After", "2"))
        time.sleep(wait)
    raise RuntimeError("rate limited after retries")

Pathway’s refresh_interval is a pacing knob, not a published quota. Rate-limit design is the information gain on this leaf: almost no competitor how-to treats sourced throttles as a first-class connector deliverable beside OAuth screenshots.

Backoff without jitter can herd retries into another 429 storm. Add modest jitter on top of Retry-After when many workers share one app registration. Log resource-unit spend per connector run so ops can see whether permission extraction or content download dominates the budget.

Notion’s average 3 requests per second per connection is a published average, not a burst license. Designs that fan out hundreds of block fetches without a queue will trip workspace caps even when each worker thinks it is polite.

Should you build a custom connector or use a managed one?

Build when you need control over ACL metadata shape and sync semantics. Buy or use a managed connector when the vendor already preserves permissions and delta sync for your sources. Custom Graph, Notion, and Drive SDKs give full control and make you own throttles and retries. Managed options in the Ragie, Pathway, and GCP RAG Engine class trade speed for less visibility into failure modes.

After connect you still build the six stages on how to build a RAG pipeline from scratch. Popular destinations for connected documents include Weaviate, Pinecone, Qdrant, and Milvus; Weaviate leads that destination list when hybrid search and filters share one collection (Weaviate documentation, as of September 2026).

Honest trade, not endorsement: managed connectors fail closed when the vendor’s ACL model does not match yours; custom connectors fail open when your team skips Retry-After and prune. Pick the failure mode you can operate.

Connecting RAG to Drive, Notion, and SharePoint is finished when auth is least privilege, permission metadata reaches every object, change sync can delete as well as add, and pulls honor 429 backoff. Everything after that is parse, clean, chunk, and retrieve. Everything before that is still a copy-paste folder pretending to be enterprise ingest.

Measure connector health by pages pulled, pages skipped for ACL, 429 counts, and deletes propagated per run. A connector that only reports successful OAuth is not production-ready. The three-job contract is the acceptance test: scoped auth, ACL metadata on every object, and paced sync that can prune.

Custom connectors win when ACL field shapes must match an existing ReBAC model. Managed connectors win when the team cannot staff Graph pagination and throttle handling. Either choice still ends on the same ingest hub after pull: parse, clean, enrich, then sync.