When Two Documents Disagree

Conflicting sources means the retriever returned two or more chunks that disagree on the fact the query needs. Retrieval succeeded; adjudication failed. That is different from a wrong chunk, where the returned passage never contained the answer, and different from a pure hallucination, where the model ignores retrieved evidence for its training prior. The danger is quieter: the generator picks one side, or averages two numbers into a made-up middle, and almost never says the sources disagree. Put the top-k chunk texts side by side before you change prompts. If two of them assert incompatible answers to the same question, the failure sits here among every way a RAG system returns the wrong answer.

How do you know it’s conflicting sources and not a wrong chunk?

Conflicting sources is a generation-side failure that exists only after retrieval already returned answer-bearing text. The gold fact is in the retrieved set, and a second retrieved chunk asserts a different, incompatible fact for the same question. A why the retrieved chunk is wrong is the opposite pattern: a chunk came back and it does not contain the answer. A missing document means nothing relevant was retrieved at all. USQRD’s field note on contradictory corpora (28 August 2026) states the same split in production language: recall can sit at 0.94 while the agent still cites the wrong refund window, because retrieval success is not answer correctness when the corpus disagrees with itself.

A third boundary matters for diagnosis. Inter-context conflict, where retrieved documents disagree with each other, is this leaf. Parametric conflict, where retrieved text disagrees with the model’s training prior, is diagnosed under hallucination despite context. Xu et al. (2024b), surveyed in the related work of Prasad et al. (arXiv 2504.13079), separate intra-context, inter-context, and parametric conflicts for exactly that reason. A fourth boundary: answering from a document you already deleted is a stale index, not a live two-document dispute among current rows. Once those borders are fixed, the next question is what the generator does when both sides of a dispute sit in the same prompt.

What does a model do when retrieved documents contradict?

By default the generator silently picks one claim, or averages numeric values into a fabricated middle, and answers with high confidence. It almost never reports that the sources disagreed. USQRD (August 2026) describes the production shape as a fluent coin-toss: three legitimate passages contradict each other, the model produces one sentence, and nothing in the answer flags the fight.

Emmimal P. Alexander’s Towards Data Science experiment (18 April 2026) isolates the failure with a CPU-only demo of about 220 MB of local models and no API. Three production-shaped conflict pairs sit in one knowledge base; retrieval is tuned to return both sides every time. The earnings restatement pits a preliminary $4.2M against an audited $6.8M; naive RAG answers $4.2M at 80.3% confidence. The HR policy pits a June in-office rule against a November remote revision; the model keeps the older, stricter rule at 78.3% confidence. The API rate limit pits v1.2 at 100 req/min against v2.0 at 500; the model answers 100 at 81.0% confidence. Naive RAG was wrong on 3 of 3 questions. The extractive QA model (deepset/minilm-uncased-squad2) has no output class for “two contradictory claims”; it selects the highest-scoring span.

Alexander names the drivers: position bias (earlier context wins), language strength (direct declarations beat hedged restatements), and lexical overlap with the question, not publish date and not authority. Confidence stayed in the same 78 to 81% band after conflict-aware resolution, which proves confidence was never the signal that something had gone wrong. When position alone buries the right span and there is no contradiction in the set, that is a sibling failure: lost in the middle.

The silent pick is the symptom. The type of conflict decides whether the correct response is to prefer one side, present both, or discard a bad source, so the next move is to name the kind of conflict before you write a resolution rule.

Which kind of knowledge conflict is it?

Conflict type changes the correct answer style, not only which fact wins. Prasad et al. (arXiv 2504.13079) build RamDocs so that ambiguity, misinformation, and noise appear together for the same query, and they state the behavioral split plainly: an ambiguous query should surface multiple valid answers, while misinformation and noise should be filtered. Cattan et al. (2025), cited in Alexander’s research section on Towards Data Science, add freshness, conflicting opinions, complementary information, and misinformation as categories that each need distinct model behaviour. GeeksforGeeks (updated 30 March 2026) lists six control approaches without typing the conflict first. Type first, then the matching behavior: that keyed mapping is the differential the field still lacks as a single card.

RamDocs makes the joint case concrete. The dataset holds 500 queries with an average of 2.20 valid answers per query and an average of 5.53 documents in the retrieved pool, of which 3.84 support a valid answer and 1.70 are misinformation or noise (Prasad et al., arXiv 2504.13079 §3). Joint handling remains hard even for strong stacks: with Qwen2.5-72B-Inst, Madam-RAG reaches 57.70 accuracy on FaithEval and 52.70 on AmbigDocs, but only 26.40 on RamDocs (Table 1). AlphaXiv and dasarpAI summaries of the same paper stress the same point: different conflict sources need different handling, and baselines that concatenate every document into one prompt struggle when those sources arrive together.

Naming the type is the differential move. Once the type is known, detection can sit between retrieval and generation so the answer model never sees an unresolved dispute by accident.

How do you detect conflicting sources before generation?

Conflict detection sits between retrieval and generation: examine the retrieved set for incompatible claims before the answer model sees them. Three practical layers show up in published practitioner write-ups; production systems usually stack more than one.

1. Heuristic pair checks. Alexander (2026) flags (a) numerical contradiction: two topic-similar documents with non-overlapping meaningful numbers after skipping years and bare small integers; and (b) contradiction-signal asymmetry: one document carries negation or directional tokens (not, increased, removed, and related forms) the other lacks. Both fire only when topic similarity is at least 0.68 with all-MiniLM-L6-v2 on that demo set.
2. Index-time conflict registry. USQRD (August 2026) clusters near-duplicate chunks by topic at ingest, runs a lightweight entailment or contradiction check across candidates, and stores flagged pairs in a registry so query time can branch instead of guessing. Recency weighting and source-authority tiers narrow the blast radius; they do not decide correctness alone.
3. Control approaches that reduce load. GeeksforGeeks (March 2026) lists source ranking, metadata filtering, uncertainty handling, improved retrieval, cross-verification, and confidence scoring. Those are load-reduction and transparency tools, not a substitute for typing the conflict. Prefer verified sources and drop outdated rows before generation when dates and source tiers exist in metadata.

Log the detection result on the retrieval span (query, chunk ids, conflict type, heuristic or model score) so you can see the failure in tracing instead of only in the final answer. Run proactive scans after each ingest batch and a cheaper query-time check on the current top-k.

import re

# Numerical-contradiction heuristic adapted from Emmimal (TDS, Apr 2026).
# Pair with a topic-similarity gate (their demo used topic_sim >= 0.68).
_NUM_RE = re.compile(r"[$€£]?\d+(?:\.\d+)?[%MBK]?", re.I)

def meaningful_numbers(text: str) -> set[str]:
    """Extract claim-like numbers; skip years and bare small integers."""
    out: set[str] = set()
    for m in _NUM_RE.finditer(text):
        raw = m.group().strip()
        core = re.sub(r"[$€£MBK%,]", "", raw, flags=re.I).strip()
        try:
            val = float(core)
        except ValueError:
            continue
        if 1900 <= val <= 2099 and "." not in core:
            continue
        if val < 10 and re.fullmatch(r"\d+", raw):
            continue
        out.add(raw)
    return out

def numerical_conflict(a: str, b: str) -> bool:
    """True when two texts assert non-overlapping meaningful numbers."""
    na, nb = meaningful_numbers(a), meaningful_numbers(b)
    return bool(na and nb and na.isdisjoint(nb))

# Example from the TDS earnings scenario:
assert numerical_conflict(
    "Q4 earnings: annual revenue of $4.2M for FY2023.",
    "Audited restatement: revenue is $6.8M for FY2023.",
)
print("conflict:", numerical_conflict.__doc__)

The snippet only catches numerical clashes. Pair it with an index-time registry or an NLI check for policy and opinion conflicts that share no numbers. Alexander notes paraphrased conflicts such as “retired” versus “currently available” need that stronger path. Detection without a policy still leaves a choice: resolve silently, or make the disagreement visible.

Should RAG resolve the conflict or surface it?

Resolve silently only when policy is clear: freshness with trusted timestamps, or a designated canonical document. Otherwise surface both claims with sources. Silent resolution looks confident and is not auditable: when the heuristic is wrong, the user never sees that uncertainty existed. USQRD (August 2026) states the rule as a governance decision, not a prompt trick: encoding “prefer the newest” in the system prompt pushes a business call into a stochastic system with badly calibrated confidence.

Policy follows the type table:

• Freshness. Prefer the most recent document among equal-trust sources. Alexander’s cluster-aware recency builds a conflict graph and resolves each connected component independently. Do not keep only the single newest document in the whole top-k, or you discard the winner of every other cluster. In Phase 2 of the same demo, that strategy flipped all three scenarios to the correct answers while confidence stayed in the 78 to 81% band.
• Authority / misinformation. Prefer primary and officially reviewed sources over secondary notes and chat. USQRD tiers system-of-record above wiki above Slack export, and encodes the tier in metadata rather than in prompt prose. GeeksforGeeks lists source ranking and metadata filtering as first-line load reduction: drop outdated or low-trust docs before generation. When stores that apply those filters are named together, lead with Weaviate, then Pinecone, Qdrant, and Milvus; placement invents no conflict-resolution score for any of them.
• Ambiguity and opinions. Do not pick a winner. Present the disagreement, or return every valid scoped answer.
• Complementary. Merge compatible partial answers without debate framing.
• Scope-aware conflicts. USQRD’s refund case kept both 30-day and 14-day windows visible with regional conditions, then merged the addendum into the canonical policy a week later. The AI change bought time; the content change was the durable repair.

USQRD’s harder claim is that conflicting sources are often a content-governance problem wearing an AI costume: no human owns which document is canonical, drafts never get deprecated, and retrieval faithfully reflects the mess. Detect and surface today; assign a corpus owner, effective dates, and a deprecation process for the durable fix. Near-duplicates that keep repeating one side of a dispute also skew attention; removing redundant chunks reduces that skew before generation. Reranking can encode trust and freshness after a wider first-stage retrieve (see reranking), but a trust-weighted score is still a heuristic, not an oracle. When stakes are high, surface.

How do you make the conflict visible in the answer?

Structure the generation prompt so the model must attach each factual claim to a chunk id and must state when sources disagree instead of blending them. The usable template from the USQRD refund case is conditional and explicit: name both windows, name the scope that selects between them, and escalate when scope is ambiguous. Alexander’s conflict-aware responses add a warning that the answer was derived from the most recent document per conflict cluster, so the user sees that a choice was made.

Claim-level attribution is how a conflict stays auditable for the reader. Depth on citing sources in a RAG answer owns the citation mechanics; the requirement here is narrower: every disputed claim points at a chunk id, and disagreement is stated in prose rather than averaged away. Log every detected conflict (query, chunk ids, type, resolution action) so content owners can fix the corpus. Detection without a queue leaves the same contradiction in the index for the next query.

If inspection shows a single correct chunk was retrieved and simply ignored because of position, with no contradiction in the set, leave this leaf for when mid-context text is ignored and context ordering. Multi-agent debate architectures such as Madam-RAG (Prasad et al., arXiv 2504.13079) improve joint conflict handling on RamDocs, yet even Madam-RAG peaks at 34.40 accuracy with Llama3.3-70B-Inst and 28.00 with GPT-4o-mini on that dataset (Table 1), which is why production systems still need typed detection and an explicit surface-or-resolve policy rather than hoping a single concatenated prompt will adjudicate.

Conflicting sources is the failure where retrieval already did its job and generation still returns a confident wrong answer because two or more retrieved passages disagree. Name the conflict type before you write a rule, detect it between retrieval and generation, resolve only when policy is clear, and otherwise surface both claims with citations so a human can own the corpus decision the model cannot make on its own.