When Two Documents Disagree

Conflicting sources means the retriever returned two or more chunks that disagree on the fact your query needs. Retrieval succeeded; adjudication failed. This is not a wrong chunk (the answer-bearing text never came back) and not a pure hallucination (the model ignoring context for its training prior). The danger is quieter: the model picks one side — or averages two numbers into a made-up middle — and almost never says the sources disagree.

Before you change the prompt, put the top-k chunk texts side by side. If two of them assert incompatible answers to the same question, you are on this page. If none of them contains the answer, leave — that is retrieval.

How do you know it’s conflicting sources and not a wrong chunk?

Conflicting sources is a generation-side failure that only exists after retrieval already returned answer-bearing text. The gold fact is in the retrieved set, and a second retrieved chunk asserts a different, incompatible fact for the same question. A wrong chunk is the opposite pattern: a chunk came back and it does not contain the answer. A missing document means nothing relevant was retrieved at all.

A third boundary matters: context–context conflict (documents disagree with each other) is this page; context–memory conflict (retrieved text disagrees with the model’s parametric prior) is diagnosed under hallucination despite context — Gokul, Tenneti and Nakkiran (2025, arXiv:2504.00180) draw that split explicitly. Chen et al. (2026, arXiv:2605.14473) study the related question of whether the model follows retrieved evidence when it conflicts with prior knowledge; that compliance problem is not “two docs disagree.” A fourth: answering from a document you already deleted is a stale index, not a live two-document dispute.

What does a model do when retrieved documents contradict?

By default the generator silently picks one claim — or averages numeric values into a fabricated middle — and answers with high confidence. It almost never reports that the sources disagreed (TypeGraph, undated production write-up on contradiction detection; aiworldinformation system-design note, reviewed July 2026).

Emmimal P. Alexander’s Towards Data Science experiment (18 April 2026) isolates the failure with a CPU-only demo (~220 MB of local models, no API). Three production-shaped conflict pairs sit in one knowledge base; retrieval is tuned to return both sides every time:

• Earnings restatement: preliminary $4.2M vs audited $6.8M — naive answer $4.2M at 80.3% confidence.
• HR policy: June in-office rule vs November remote revision — model keeps the older, stricter rule at 78.3% confidence.
• API rate limit: v1.2 at 100 req/min vs v2.0 at 500 — model answers 100 at 81.0% confidence.

Naive RAG was wrong on 3 of 3 questions. The extractive QA model (deepset/minilm-uncased-squad2) has no output class for “two contradictory claims”; it selects the highest-scoring span. Alexander names the drivers: position bias (earlier context wins), language strength (direct declarations beat hedged restatements), and lexical overlap with the question — not publish date and not authority. When position alone buries the right span and there is no contradiction, that is a sibling failure — lost in the middle.

Which kind of knowledge conflict is it?

Conflict type changes the correct answer style, not only which fact wins. Cattan et al. / Google Research (2025, arXiv:2506.08500, “DRAGged into Conflicts”) annotate a Conflicts benchmark of 458 queries with expert conflict-type labels and define an expected model behavior per type. Vanilla RAG prompts on that benchmark reach only 59.4–68.3% expected-behavior adherence across the models they report; a pipeline that first predicts the conflict type then generates gains about 9 points on average; an oracle that receives the gold type gains about 24 points — while answer recall and factual grounding stay high. Naming the type is the differential move.

GeeksforGeeks (updated March 2026) lists six control approaches — source ranking, metadata filtering, uncertainty handling, better retrieval, cross-verification, confidence scoring — without typing the conflict first. Towards Data Science resolves detected conflicts by cluster-aware recency only. TypeGraph separates temporal updates from overlapping contradictions via validity windows. The table above is the missing link: type first, then the matching behavior.

How do you detect conflicting sources before generation?

Conflict detection sits between retrieval and generation: examine the retrieved set for incompatible claims before the answer model sees them. Three layers show up across the live ranking pages; production systems usually stack more than one.

1. Heuristic pair checks. Alexander (2026) flags (a) numerical contradiction — two topic-similar documents with non-overlapping meaningful numbers after skipping years and bare small integers — and (b) contradiction-signal asymmetry — one document carries negation or directional tokens (not, increased, removed, …) the other lacks. Both fire only when topic similarity is at least 0.68 with all-MiniLM-L6-v2 on that demo set.
2. LLM as context validator. Gokul et al. (2025) ask the model three jobs on a retrieved set: is there a conflict, what type, which documents. On their HotpotQA-derived synthetic set (1,867 samples), Claude-3 Sonnet with chain-of-thought reaches conflict-detection accuracy 0.710 and F1 0.710; pair contradictions are easiest (Llama-3.3 70B basic accuracy 0.893); self-contradictions inside one document are hardest (accuracies reported as low as 0.006–0.456). CoT helps Claude and can hurt Llama — strategy is model-dependent.
3. Structured claims. Align subject–predicate–object triples across chunks; same subject and predicate with different objects is a candidate conflict. Non-overlapping temporal validity windows are updates, not contradictions; overlapping windows are genuine conflicts (TypeGraph). Run proactive scans after each ingest batch and a cheaper query-time check on the current top-k.

Log the detection result on the retrieval span — query, chunk ids, conflict type, heuristic or model score — so you can see the failure in tracing instead of only in the final answer.

import re

# Numerical-contradiction heuristic adapted from Emmimal (TDS, Apr 2026).
# Pair with a topic-similarity gate (their demo used topic_sim >= 0.68).
_NUM_RE = re.compile(r"[$€£]?d+(?:.d+)?[%MBK]?", re.I)

def meaningful_numbers(text: str) -> set[str]:
    """Extract claim-like numbers; skip years and bare small integers."""
    out: set[str] = set()
    for m in _NUM_RE.finditer(text):
        raw = m.group().strip()
        core = re.sub(r"[$€£MBK%,]", "", raw, flags=re.I).strip()
        try:
            val = float(core)
        except ValueError:
            continue
        if 1900 <= val <= 2099 and "." not in core:
            continue
        if val < 10 and re.fullmatch(r"d+", raw):
            continue
        out.add(raw)
    return out

def numerical_conflict(a: str, b: str) -> bool:
    """True when two texts assert non-overlapping meaningful numbers."""
    na, nb = meaningful_numbers(a), meaningful_numbers(b)
    return bool(na and nb and na.isdisjoint(nb))

# Example from the TDS earnings scenario:
assert numerical_conflict(
    "Q4 earnings: annual revenue of $4.2M for FY2023.",
    "Audited restatement: revenue is $6.8M for FY2023.",
)
print("conflict:", numerical_conflict.__doc__)

The snippet only catches numerical clashes. Pair it with an LLM validator (or SPO alignment) for policy and opinion conflicts that share no numbers.

Should RAG resolve the conflict or surface it?

Resolve silently only when policy is clear — freshness with trusted timestamps, or a designated canonical document. Otherwise surface both claims with sources. Silent resolution looks confident and is not auditable: when the heuristic is wrong, the user never sees that uncertainty existed (TypeGraph).

Policy follows the type table:

• Freshness. Prefer the most recent document among equal-trust sources. Alexander’s cluster-aware recency builds a conflict graph and resolves each connected component independently — do not keep only the single newest document in the whole top-k, or you discard the winner of every other cluster.
• Authority / misinformation. Prefer primary and officially reviewed sources over secondary notes and chat (paulserban trust levels; TypeGraph source_type × recency × author_authority). GeeksforGeeks lists source ranking and metadata filtering as first-line load reduction: drop outdated or low-trust docs before generation.
• Opinions. Do not pick a winner. Present the disagreement.
• Complementary. Merge compatible partial answers without debate framing.

Reranking can encode trust and freshness after a wider first-stage retrieve — see reranking — but a trust-weighted score is still a heuristic, not an oracle. When stakes are high, surface.

How do you make the conflict visible in the answer?

Structure the generation prompt so the model must attach each factual claim to a chunk id and must state when sources disagree instead of blending them. TypeGraph’s pattern is the usable template: report both versions with dates and sources, state which is more likely under your policy, and invite verification.

Cattan et al. instruct models to emit inline citations and score factual grounding as the share of supported sentences — the same discipline that makes a surfaced conflict auditable. A pipeline or taxonomy-aware prompt (predict type, then generate) is the DRAGged Result 2–3 improvement path when you can afford the extra call.

Log every detected conflict — query, chunk ids, type, resolution action — so content owners can fix the corpus. Detection without a queue leaves the same contradiction in the index for the next query (aiworldinformation; TypeGraph resolution workflow).

If inspection shows a single correct chunk was retrieved and simply ignored because of position, with no contradiction in the set, leave this page for lost in the middle and context ordering. The failures hub at /failures maps each symptom to its stage.