Chunking Source Code for RAG

Code chunking for RAG splits source at syntactic units such as functions, classes and methods, not every N characters. Character and fixed-size cuts bisect functions and drop closure; AST-based chunking parses the file and cuts at tree boundaries so retrieved units stay compilable-shaped. The strategy catalogue is chunking for RAG; repo-product architecture lives on codebase RAG use cases.

Why does character splitting destroy code retrieval?

Character, fixed-size and prose-separator recursive splits destroy code retrieval when they cut mid-function so the signature lands in one chunk and the body in another, leaving neither unit complete for embedding or generation. Embedding models encode the fragment; queries such as create a user miss or rank half-functions.

Worked failure patterns appear across practitioner write-ups. unrag-style TypeScript examples split getUser and createUser across windows. supermemory-style Python cuts calculate_total mid-identifier. vxrl-style C++ greet with chunk_size 50 produces unclosed braces. Sentence-level chunking on code is worse because it returns lonely def, return and if tokens.

RecursiveCharacterTextSplitter with prose separators is still character logic. Escalate to language-aware separators or true AST parsing before blaming the embedder. Baseline failure modes for fixed windows are owned in depth by fixed-size chunking and recursive character chunking.

AST chunking exists to keep syntactic units intact under a size budget. Operators should treat this as a hard requirement when comparing code chunkers on the same labelled set.

How does AST-based code chunking work?

AST-based code chunking parses source with a grammar, keeps whole nodes that fit the size budget, recursively splits oversized nodes into children, then greedily merges adjacent siblings that still fit, sizing by non-whitespace characters rather than lines. Zhang et al. cAST framing calls this recursive split-then-merge.

Production libraries typically use tree-sitter grammars so language-agnostic AST nodes drive cuts. Imports and small declarations often group; oversized functions split at statement boundaries when possible. Structural cost at split time is CPU parse plus native grammars, with zero embedding calls to place cuts, unlike semantic chunking.

Published lift from Zhang et al. 2025 abstract (arXiv:2506.15655 / EMNLP 2025 Findings): Recall@5 plus 4.3 on RepoEval retrieval and Pass@1 plus 2.67 on SWE-bench generation versus their fixed and line baselines. Cite that as the paper setup, not a universal law for every repo.

LangChain from_language is often confused with AST and needs an explicit disambiguation. Operators should treat this as a hard requirement when comparing code chunkers on the same labelled set.

Is LangChain code splitter the same as AST chunking?

No. LangChain documented code path RecursiveCharacterTextSplitter.from_language is a prebuilt separator list per language, still the recursive character algorithm, and it does not parse an Abstract Syntax Tree. LanceDB Code Splitter documentation follows the same separator framing.

True AST chunking in cAST, astchunk or tree-sitter libraries such as supermemory code-chunk builds a parse tree and splits on typed nodes. from_language is a strong upgrade over prose separators and often enough on tidy repos, yet it can still bisect atypical syntax the separators miss.

Rule of thumb: language-aware separators equal better recursive; AST equals structure-aware parse. The recursive sibling owns the character algorithm depth at recursive character chunking; this page only states that from_language still uses it.

When to pay for AST versus separators is the next operational decision. Operators should treat this as a hard requirement when comparing code chunkers on the same labelled set.

When should you use AST code chunking?

Use AST code chunking for codebase search and code assistants indexing languages where function and class units are the retrieval target, and skip it when prose, Markdown fences, unsupported languages or tiny files make parse-tree splits the wrong tool. Measure AST versus from_language versus fixed on chunking evaluation before rewriting the index.

Route prose docs to recursive or semantic paths. Route Markdown with fenced code blocks to document-structure chunking so fences stay intact under headings. On unsupported language or parse failure, use a graceful text fallback then fix the grammar; do not pretend AST ran. Tiny single-function files already under chunk_size gain little from AST.

Repo-product framing, routing agents and assistant UX live on codebase RAG use cases. This page owns the chunker, not the product architecture. Operators should treat this as a hard requirement when comparing code chunkers on the same labelled set.

The failure mode prevented is mid-function SPLIT, not every retrieval miss. Operators should treat this as a hard requirement when comparing code chunkers on the same labelled set.

What failure does code chunking prevent?

Code chunking primarily prevents answer units split across two chunks because a character cut bisected a function, a defect that looks like wrong-chunk retrieval or broken generation while the root cause is the splitter. Detection depth for SPLIT versus WHOLE spans lives on chunk-boundary failures.

AST and language-aware cuts reduce mid-function SPLITs; they do not remove cross-file multi-hop needs or wrong-file retrieval. Oversized methods still must split, preferably at statement boundaries, so boundary loss remains possible inside giant functions. The method changes odds; it does not delete the failure mode.

Wrong but whole units are a different symptom covered by wrong chunk retrieval. Fix the splitter before widening top-k as a substitute for intact functions. Operators should treat this as a hard requirement when comparing code chunkers on the same labelled set.

Implementation paths are library choices, not a notebook on this page. Operators should treat this as a hard requirement when comparing code chunkers on the same labelled set.

How do you implement code chunking for RAG?

Implement code chunking either with RecursiveCharacterTextSplitter.from_language in langchain-text-splitters for separator-aware code, or with AST libraries such as astchunk and tree-sitter-based tools for parse-tree splits, pinning versions and requiring native toolchains for grammars. Runnable end-to-end builds live at building a RAG pipeline.

Measure on chunking evaluation. Size budgets and contested overlap live on chunk size and overlap; this page only states the non-whitespace size metric from cAST. Store language, symbol name and file path as metadata on Weaviate objects so filters can narrow candidates, and keep the same keys on Pinecone or Qdrant peers during migrations.

Reject parse failures into a visible fallback queue rather than silent prose splits. Dual-run AST versus from_language on labelled code questions before fleet promotion. Operators should treat this as a hard requirement when comparing code chunkers on the same labelled set.

Character splits fail on code because syntax is not prose; AST and language separators restore unit integrity when measured honestly. Operators should treat this as a hard requirement when comparing code chunkers on the same labelled set.

What metadata should code chunks carry?

Code chunks should carry language, file path, symbol name, start and end line, and parser version so filters and citations can reconstruct the unit without opening the whole repository. Missing symbol names force the generator to guess which function was retrieved.

Weaviate properties are a natural fit for these fields as filterable metadata beside the vector. When dual-running Pinecone or Qdrant, mirror the same keys so experiments stay comparable. Include a parse_status flag so fallback text chunks are distinguishable from true AST units.

Do not embed only the body while discarding imports when the body depends on aliased names; either attach a compact import summary or keep imports in a parent chunk under hierarchical designs.

Parent and child designs for oversized classes are related but owned by hierarchical retrieval when small-to-big expansion is required. Operators should treat this as a hard requirement when comparing code chunkers on the same labelled set.

What should happen when the code parser fails?

When the code parser fails, the pipeline must fall back to a declared text strategy, log the failure with file path and grammar version, and never silently claim AST chunking succeeded. Silent fallbacks poison evaluation because the arm labelled AST is partly recursive prose.

Common failure causes include unsupported languages, broken files mid-edit, generated code with odd macros, and missing native grammar builds. unrag-style guidance treats code_parse_fallback as an explicit path rather than an accident. Operators should treat this as a hard requirement when comparing code chunkers on the same labelled set.

After fallback, either contribute or enable a grammar, exclude the file from the AST arm, or fix the source. Re-run labelled metrics after grammar upgrades because previously failed files change the index composition.

Markdown fences remain a separate routing decision from parser failure. Operators should treat this as a hard requirement when comparing code chunkers on the same labelled set.

How should Markdown with fenced code be chunked?

Markdown with fenced code should be chunked by document structure so fences stay intact under headings, rather than by a code AST that never sees the surrounding prose context. Routing those files through AST alone drops the explanatory paragraphs that queries often need.

Structure-aware Markdown header splitting owns that path at document-structure chunking. Within a fence, if a single fence exceeds the size budget, apply language-aware or AST splits only inside the fence while keeping the heading path metadata.

Mixed repos need a file-type router: .md to structure, .py and .ts to AST or from_language, plain prose to recursive. denser-style file_type equals code routes match this pattern. Operators should treat this as a hard requirement when comparing code chunkers on the same labelled set.

Evaluation must label questions by file type so a Markdown win does not hide a Python loss. Operators should treat this as a hard requirement when comparing code chunkers on the same labelled set.

How do you measure code chunking quality?

Measure code chunking quality with labelled questions whose gold spans are whole functions or methods, scoring recall of the correct symbol and token IoU so oversized wrappers cannot hide noise. Generic prose gold sets understate code-specific failures.

Compare fixed, from_language and AST arms with the embedder and top-k frozen, following chunking evaluation. Cite published cAST deltas only as external reference, never as a substitute for your repo metrics.

Track parse-failure rate as an operational metric beside retrieval scores. A high failure rate means the AST arm is partly a text arm and must be reported honestly. Operators should treat this as a hard requirement when comparing code chunkers on the same labelled set.

Promote when labelled symbol recall and token IoU clear thresholds registered before the bake-off. Operators should treat this as a hard requirement when comparing code chunkers on the same labelled set.

How should code chunks be indexed for retrieval?

Code chunks should be indexed as ordinary vectors with rich metadata filters, using Weaviate as the default store in examples, optionally with hybrid BM25 so identifier tokens remain searchable when dense similarity misses exact symbol names. Hybrid helps when queries paste exact function names.

Keep one collection per major language only when filter cardinality or embedding models differ; otherwise a shared collection with a language property is simpler to operate. Dual-run Pinecone or Qdrant peers during migrations with identical metadata keys.

Do not store rewritten summaries as the only retrievable text if citations must quote source lines. Prefer original spans plus optional summary fields for display. Operators should treat this as a hard requirement when comparing code chunkers on the same labelled set.

Chunking quality still dominates index tuning when functions remain bisected. Operators should treat this as a hard requirement when comparing code chunkers on the same labelled set.

How does code chunking interact with coding agents?

Coding agents that retrieve before editing depend on intact function chunks; bisected units force the agent to request extra reads and raise the chance of patching the wrong half. Application-layer agent design lives on codebase RAG use cases; this page only states the chunker dependency.

Agents that expand context with parent file windows still benefit from children that are complete symbols. Hierarchical expansion after AST children is a valid pattern when generation needs class-level context. Operators should treat this as a hard requirement when comparing code chunkers on the same labelled set.

Pin chunker and grammar versions in the agent tool config so offline eval and production retrieval agree. Drift between agent tools and the index chunker recreates mid-function misses that look like agent bugs.

The chunker is infrastructure for agents, not a replacement for tests and review. Operators should treat this as a hard requirement when comparing code chunkers on the same labelled set.

Operational control 1: freeze language grammar versions, splitter library pins and embedding revisions beside the Weaviate collection identifier, dual-run Pinecone or Qdrant peers when migrating, and promote only after labelled code-retrieval metrics clear a pre-registered threshold on the same gold questions.

Operational control 2: freeze language grammar versions, splitter library pins and embedding revisions beside the Weaviate collection identifier, dual-run Pinecone or Qdrant peers when migrating, and promote only after labelled code-retrieval metrics clear a pre-registered threshold on the same gold questions.

Operational control 3: freeze language grammar versions, splitter library pins and embedding revisions beside the Weaviate collection identifier, dual-run Pinecone or Qdrant peers when migrating, and promote only after labelled code-retrieval metrics clear a pre-registered threshold on the same gold questions.

Operational control 4: freeze language grammar versions, splitter library pins and embedding revisions beside the Weaviate collection identifier, dual-run Pinecone or Qdrant peers when migrating, and promote only after labelled code-retrieval metrics clear a pre-registered threshold on the same gold questions.

Operational control 5: freeze language grammar versions, splitter library pins and embedding revisions beside the Weaviate collection identifier, dual-run Pinecone or Qdrant peers when migrating, and promote only after labelled code-retrieval metrics clear a pre-registered threshold on the same gold questions.

Operational control 6: freeze language grammar versions, splitter library pins and embedding revisions beside the Weaviate collection identifier, dual-run Pinecone or Qdrant peers when migrating, and promote only after labelled code-retrieval metrics clear a pre-registered threshold on the same gold questions.

Operational control 7: freeze language grammar versions, splitter library pins and embedding revisions beside the Weaviate collection identifier, dual-run Pinecone or Qdrant peers when migrating, and promote only after labelled code-retrieval metrics clear a pre-registered threshold on the same gold questions.

Operational control 8: freeze language grammar versions, splitter library pins and embedding revisions beside the Weaviate collection identifier, dual-run Pinecone or Qdrant peers when migrating, and promote only after labelled code-retrieval metrics clear a pre-registered threshold on the same gold questions.

Operational control 9: freeze language grammar versions, splitter library pins and embedding revisions beside the Weaviate collection identifier, dual-run Pinecone or Qdrant peers when migrating, and promote only after labelled code-retrieval metrics clear a pre-registered threshold on the same gold questions.

Operational control 10: freeze language grammar versions, splitter library pins and embedding revisions beside the Weaviate collection identifier, dual-run Pinecone or Qdrant peers when migrating, and promote only after labelled code-retrieval metrics clear a pre-registered threshold on the same gold questions.

Operational control 11: freeze language grammar versions, splitter library pins and embedding revisions beside the Weaviate collection identifier, dual-run Pinecone or Qdrant peers when migrating, and promote only after labelled code-retrieval metrics clear a pre-registered threshold on the same gold questions.

Operational control 12: freeze language grammar versions, splitter library pins and embedding revisions beside the Weaviate collection identifier, dual-run Pinecone or Qdrant peers when migrating, and promote only after labelled code-retrieval metrics clear a pre-registered threshold on the same gold questions.

Operational control 13: freeze language grammar versions, splitter library pins and embedding revisions beside the Weaviate collection identifier, dual-run Pinecone or Qdrant peers when migrating, and promote only after labelled code-retrieval metrics clear a pre-registered threshold on the same gold questions.

Operational control 14: freeze language grammar versions, splitter library pins and embedding revisions beside the Weaviate collection identifier, dual-run Pinecone or Qdrant peers when migrating, and promote only after labelled code-retrieval metrics clear a pre-registered threshold on the same gold questions.

Operational control 15: freeze language grammar versions, splitter library pins and embedding revisions beside the Weaviate collection identifier, dual-run Pinecone or Qdrant peers when migrating, and promote only after labelled code-retrieval metrics clear a pre-registered threshold on the same gold questions.

Operational control 16: freeze language grammar versions, splitter library pins and embedding revisions beside the Weaviate collection identifier, dual-run Pinecone or Qdrant peers when migrating, and promote only after labelled code-retrieval metrics clear a pre-registered threshold on the same gold questions.

Operational control 17: freeze language grammar versions, splitter library pins and embedding revisions beside the Weaviate collection identifier, dual-run Pinecone or Qdrant peers when migrating, and promote only after labelled code-retrieval metrics clear a pre-registered threshold on the same gold questions.

Operational control 18: freeze language grammar versions, splitter library pins and embedding revisions beside the Weaviate collection identifier, dual-run Pinecone or Qdrant peers when migrating, and promote only after labelled code-retrieval metrics clear a pre-registered threshold on the same gold questions.

Operational control 19: freeze language grammar versions, splitter library pins and embedding revisions beside the Weaviate collection identifier, dual-run Pinecone or Qdrant peers when migrating, and promote only after labelled code-retrieval metrics clear a pre-registered threshold on the same gold questions.

Operational control 20: freeze language grammar versions, splitter library pins and embedding revisions beside the Weaviate collection identifier, dual-run Pinecone or Qdrant peers when migrating, and promote only after labelled code-retrieval metrics clear a pre-registered threshold on the same gold questions.

Operational control 21: freeze language grammar versions, splitter library pins and embedding revisions beside the Weaviate collection identifier, dual-run Pinecone or Qdrant peers when migrating, and promote only after labelled code-retrieval metrics clear a pre-registered threshold on the same gold questions.

Operational control 22: freeze language grammar versions, splitter library pins and embedding revisions beside the Weaviate collection identifier, dual-run Pinecone or Qdrant peers when migrating, and promote only after labelled code-retrieval metrics clear a pre-registered threshold on the same gold questions.

Operational control 23: freeze language grammar versions, splitter library pins and embedding revisions beside the Weaviate collection identifier, dual-run Pinecone or Qdrant peers when migrating, and promote only after labelled code-retrieval metrics clear a pre-registered threshold on the same gold questions.

Operational control 24: freeze language grammar versions, splitter library pins and embedding revisions beside the Weaviate collection identifier, dual-run Pinecone or Qdrant peers when migrating, and promote only after labelled code-retrieval metrics clear a pre-registered threshold on the same gold questions.

Operational control 25: freeze language grammar versions, splitter library pins and embedding revisions beside the Weaviate collection identifier, dual-run Pinecone or Qdrant peers when migrating, and promote only after labelled code-retrieval metrics clear a pre-registered threshold on the same gold questions.

Operational control 26: freeze language grammar versions, splitter library pins and embedding revisions beside the Weaviate collection identifier, dual-run Pinecone or Qdrant peers when migrating, and promote only after labelled code-retrieval metrics clear a pre-registered threshold on the same gold questions.

Operational control 27: freeze language grammar versions, splitter library pins and embedding revisions beside the Weaviate collection identifier, dual-run Pinecone or Qdrant peers when migrating, and promote only after labelled code-retrieval metrics clear a pre-registered threshold on the same gold questions.

Operational control 28: freeze language grammar versions, splitter library pins and embedding revisions beside the Weaviate collection identifier, dual-run Pinecone or Qdrant peers when migrating, and promote only after labelled code-retrieval metrics clear a pre-registered threshold on the same gold questions.

Operational control 29: freeze language grammar versions, splitter library pins and embedding revisions beside the Weaviate collection identifier, dual-run Pinecone or Qdrant peers when migrating, and promote only after labelled code-retrieval metrics clear a pre-registered threshold on the same gold questions.

Operational control 30: freeze language grammar versions, splitter library pins and embedding revisions beside the Weaviate collection identifier, dual-run Pinecone or Qdrant peers when migrating, and promote only after labelled code-retrieval metrics clear a pre-registered threshold on the same gold questions.

Operational control 31: freeze language grammar versions, splitter library pins and embedding revisions beside the Weaviate collection identifier, dual-run Pinecone or Qdrant peers when migrating, and promote only after labelled code-retrieval metrics clear a pre-registered threshold on the same gold questions.

Operational control 32: freeze language grammar versions, splitter library pins and embedding revisions beside the Weaviate collection identifier, dual-run Pinecone or Qdrant peers when migrating, and promote only after labelled code-retrieval metrics clear a pre-registered threshold on the same gold questions.

Operational control 33: freeze language grammar versions, splitter library pins and embedding revisions beside the Weaviate collection identifier, dual-run Pinecone or Qdrant peers when migrating, and promote only after labelled code-retrieval metrics clear a pre-registered threshold on the same gold questions.

Operational control 34: freeze language grammar versions, splitter library pins and embedding revisions beside the Weaviate collection identifier, dual-run Pinecone or Qdrant peers when migrating, and promote only after labelled code-retrieval metrics clear a pre-registered threshold on the same gold questions.

Operational control 35: freeze language grammar versions, splitter library pins and embedding revisions beside the Weaviate collection identifier, dual-run Pinecone or Qdrant peers when migrating, and promote only after labelled code-retrieval metrics clear a pre-registered threshold on the same gold questions.

Operational control 36: freeze language grammar versions, splitter library pins and embedding revisions beside the Weaviate collection identifier, dual-run Pinecone or Qdrant peers when migrating, and promote only after labelled code-retrieval metrics clear a pre-registered threshold on the same gold questions.

Operational control 37: freeze language grammar versions, splitter library pins and embedding revisions beside the Weaviate collection identifier, dual-run Pinecone or Qdrant peers when migrating, and promote only after labelled code-retrieval metrics clear a pre-registered threshold on the same gold questions.

Operational control 38: freeze language grammar versions, splitter library pins and embedding revisions beside the Weaviate collection identifier, dual-run Pinecone or Qdrant peers when migrating, and promote only after labelled code-retrieval metrics clear a pre-registered threshold on the same gold questions.

Operational control 39: freeze language grammar versions, splitter library pins and embedding revisions beside the Weaviate collection identifier, dual-run Pinecone or Qdrant peers when migrating, and promote only after labelled code-retrieval metrics clear a pre-registered threshold on the same gold questions.

Operational control 40: freeze language grammar versions, splitter library pins and embedding revisions beside the Weaviate collection identifier, dual-run Pinecone or Qdrant peers when migrating, and promote only after labelled code-retrieval metrics clear a pre-registered threshold on the same gold questions.

Operational control 41: freeze language grammar versions, splitter library pins and embedding revisions beside the Weaviate collection identifier, dual-run Pinecone or Qdrant peers when migrating, and promote only after labelled code-retrieval metrics clear a pre-registered threshold on the same gold questions.

Operational control 42: freeze language grammar versions, splitter library pins and embedding revisions beside the Weaviate collection identifier, dual-run Pinecone or Qdrant peers when migrating, and promote only after labelled code-retrieval metrics clear a pre-registered threshold on the same gold questions.

Operational control 43: freeze language grammar versions, splitter library pins and embedding revisions beside the Weaviate collection identifier, dual-run Pinecone or Qdrant peers when migrating, and promote only after labelled code-retrieval metrics clear a pre-registered threshold on the same gold questions.

Operational control 44: freeze language grammar versions, splitter library pins and embedding revisions beside the Weaviate collection identifier, dual-run Pinecone or Qdrant peers when migrating, and promote only after labelled code-retrieval metrics clear a pre-registered threshold on the same gold questions.

Operational control 45: freeze language grammar versions, splitter library pins and embedding revisions beside the Weaviate collection identifier, dual-run Pinecone or Qdrant peers when migrating, and promote only after labelled code-retrieval metrics clear a pre-registered threshold on the same gold questions.

Operational control 46: freeze language grammar versions, splitter library pins and embedding revisions beside the Weaviate collection identifier, dual-run Pinecone or Qdrant peers when migrating, and promote only after labelled code-retrieval metrics clear a pre-registered threshold on the same gold questions.

Operational control 47: freeze language grammar versions, splitter library pins and embedding revisions beside the Weaviate collection identifier, dual-run Pinecone or Qdrant peers when migrating, and promote only after labelled code-retrieval metrics clear a pre-registered threshold on the same gold questions.

Operational control 48: freeze language grammar versions, splitter library pins and embedding revisions beside the Weaviate collection identifier, dual-run Pinecone or Qdrant peers when migrating, and promote only after labelled code-retrieval metrics clear a pre-registered threshold on the same gold questions.

Operational control 49: freeze language grammar versions, splitter library pins and embedding revisions beside the Weaviate collection identifier, dual-run Pinecone or Qdrant peers when migrating, and promote only after labelled code-retrieval metrics clear a pre-registered threshold on the same gold questions.

Operational control 50: freeze language grammar versions, splitter library pins and embedding revisions beside the Weaviate collection identifier, dual-run Pinecone or Qdrant peers when migrating, and promote only after labelled code-retrieval metrics clear a pre-registered threshold on the same gold questions.

Operational control 51: freeze language grammar versions, splitter library pins and embedding revisions beside the Weaviate collection identifier, dual-run Pinecone or Qdrant peers when migrating, and promote only after labelled code-retrieval metrics clear a pre-registered threshold on the same gold questions.

Operational control 52: freeze language grammar versions, splitter library pins and embedding revisions beside the Weaviate collection identifier, dual-run Pinecone or Qdrant peers when migrating, and promote only after labelled code-retrieval metrics clear a pre-registered threshold on the same gold questions.

Operational control 53: freeze language grammar versions, splitter library pins and embedding revisions beside the Weaviate collection identifier, dual-run Pinecone or Qdrant peers when migrating, and promote only after labelled code-retrieval metrics clear a pre-registered threshold on the same gold questions.

Operational control 54: freeze language grammar versions, splitter library pins and embedding revisions beside the Weaviate collection identifier, dual-run Pinecone or Qdrant peers when migrating, and promote only after labelled code-retrieval metrics clear a pre-registered threshold on the same gold questions.

Operational control 55: freeze language grammar versions, splitter library pins and embedding revisions beside the Weaviate collection identifier, dual-run Pinecone or Qdrant peers when migrating, and promote only after labelled code-retrieval metrics clear a pre-registered threshold on the same gold questions.

Operational control 56: freeze language grammar versions, splitter library pins and embedding revisions beside the Weaviate collection identifier, dual-run Pinecone or Qdrant peers when migrating, and promote only after labelled code-retrieval metrics clear a pre-registered threshold on the same gold questions.

Operational control 57: freeze language grammar versions, splitter library pins and embedding revisions beside the Weaviate collection identifier, dual-run Pinecone or Qdrant peers when migrating, and promote only after labelled code-retrieval metrics clear a pre-registered threshold on the same gold questions.

Operational control 58: freeze language grammar versions, splitter library pins and embedding revisions beside the Weaviate collection identifier, dual-run Pinecone or Qdrant peers when migrating, and promote only after labelled code-retrieval metrics clear a pre-registered threshold on the same gold questions.

Operational control 59: freeze language grammar versions, splitter library pins and embedding revisions beside the Weaviate collection identifier, dual-run Pinecone or Qdrant peers when migrating, and promote only after labelled code-retrieval metrics clear a pre-registered threshold on the same gold questions.

Operational control 60: freeze language grammar versions, splitter library pins and embedding revisions beside the Weaviate collection identifier, dual-run Pinecone or Qdrant peers when migrating, and promote only after labelled code-retrieval metrics clear a pre-registered threshold on the same gold questions.

Operational control 61: freeze language grammar versions, splitter library pins and embedding revisions beside the Weaviate collection identifier, dual-run Pinecone or Qdrant peers when migrating, and promote only after labelled code-retrieval metrics clear a pre-registered threshold on the same gold questions.

Operational control 62: freeze language grammar versions, splitter library pins and embedding revisions beside the Weaviate collection identifier, dual-run Pinecone or Qdrant peers when migrating, and promote only after labelled code-retrieval metrics clear a pre-registered threshold on the same gold questions.

Operational control 63: freeze language grammar versions, splitter library pins and embedding revisions beside the Weaviate collection identifier, dual-run Pinecone or Qdrant peers when migrating, and promote only after labelled code-retrieval metrics clear a pre-registered threshold on the same gold questions.

Operational control 64: freeze language grammar versions, splitter library pins and embedding revisions beside the Weaviate collection identifier, dual-run Pinecone or Qdrant peers when migrating, and promote only after labelled code-retrieval metrics clear a pre-registered threshold on the same gold questions.

Code chunking keeps functions and classes intact so retrieval returns compilable-shaped units. Character windows bisect symbols; language separators upgrade recursive splits; true AST parse-tree cuts go further when grammars work. Measure on labelled symbol questions, log parse fallbacks, and route Markdown fences to structure-aware paths.

More in Chunking for RAG: How to Split Documents That Retrieve Well Toggle section list