Why RAG Gets Numbers and Tables Wrong

RAG gets a table number wrong when the retrievable unit lost the structure that made the cell meaningful: the column header, the row identity, the exact digits, the unit or scale note, or the continuation across pages. Amit Patriwala (Medium, 6 July 2026) states the failure in one line: the table was destroyed during ingestion, long before retrieval or generation ran. VDF AI’s private-RAG tables guide (captured September 2026) calls the same pattern a pipeline problem, not a model problem, and notes that turning up the context window does not repair a shredded row. Zou et al. (2025, arXiv:2504.01346) show why that matters at scale on MultiTableQA (57,193 tables and 23,758 questions): traditional RAG that relies on semantic similarity alone under-retrieves table evidence compared with table-aware retrieval. This failure is not a generic wrong chunk, where any irrelevant passage came back, and it is not the case where the labelled cell arrived intact and the model still invented a figure, which is hallucination. The taxonomy of why RAG systems fail places the leaf among ingestion and chunking faults that surface as a wrong number, so the first move is to read the retrieved chunk text and the extraction artifact that produced it before changing prompts or embeddings.

Why does RAG get table numbers wrong when the document is indexed?

An indexed document does not guarantee that a table cell survived as a labelled fact. Patriwala’s enterprise PDF pipeline (flatten text, split chunks, embed, store) works for paragraphs and collapses for financial statements, clinical tables, insurance forms, and multi-column reports, because those documents are built around structure rather than prose. Towards Data Science’s production RAG mistakes series (captured September 2026) opens with the same parsing failure: the PDF had a table, and the parser returned a string, so grouped row labels such as Premium under Health and Premium under Dental collide into one token stream with no grouping marker.

VDF AI describes the retrieval geometry that follows. A shredded financial row such as Q3 | 14,204 | 3.1% | (812) still embeds near almost any finance question because it is dense with the right vocabulary, so the fragment retrieves well and reads badly: high recall with no meaning. Zou et al. (2025) measure the research form of that gap on MultiTableQA. Their T-RAG framework reports accuracy gains of 1.2 to 11.4 percent and recall gains of 1.5 to 12.5 percent against table-retrieval baselines, and an average downstream LLM improvement of 11.2 percent over the strongest corresponding baselines across the models they evaluated. Traditional RAG and ColBERT-style baselines performed poorly in their comparison, which they read as evidence that semantic similarity alone is insufficient for table corpora. Once the document is known to be indexed and the answer is still wrong, the practical question is which of five structure failures fired on that query.

How do you tell which of the five table causes fired?

Five distinct causes produce the same wrong-number symptom, and each has a different fix, so a stage-by-stage walk that always starts at extraction wastes time when the real fault is a missing unit token beside an intact digit. KX Systems’ table-heavy RAG guide (captured September 2026) stage-walks precise extraction, contextual enrichment, format standardization, and unified embeddings. VDF AI names three pipeline causes: the chunker destroys the table, some questions are not retrieval questions, and spreadsheets behave like applications. Neither ships one table where each row is a discriminating check that names the cause from a single labelled failure.

Read the table as a triage card for one failing query where you already know the gold cell. Fleet meters such as document-level recall will flatter a system that retrieved the right PDF and still answered from the wrong row. Worked retrieval formulas live with evaluation elsewhere; here the deltas are pass or fail on header presence, row identity, digit fidelity, unit tokens, and continuation integrity.

header_preservation asks whether the gold cell’s column header and a table or section title appear in the same chunk as the value. row_hit@k asks whether top-k contains the exact answer-bearing row id, not merely the right document. digit_match compares the normalized numeric string in the chunk to the source cell. unit_match requires unit, scale, or period tokens from the header or legend in that same chunk. continuation_integrity requires carried-forward headers when the gold row sits on a continuation page or under a merged parent label. The first cause to expand is header loss, because an orphan number passes most document-level checks while remaining unusable as an answer.

Did the retrieved chunk lose the column header?

A retrieved number without its column label is not an answer; it is an orphan value. Patriwala’s clinical example shows the failure after flatten-and-chunk: asking what grade Sarah received fails when one chunk holds SarahBCR and another holds Grade1David, because the relationship between Sarah and Grade equals 1 no longer exists. VDF AI’s finance row is the same geometry: without the header row the figures are meaningless, and without the units note they are ambiguous between thousands and millions.

Parth Sakpal’s LinkedIn walkthrough of tables versus text in RAG (captured September 2026) describes the legacy flatten format and the issues that follow when newer models still receive misaligned headers and values. Once headers travel with every row, the next failure to rule out is retrieving a neighbour row from the same document.

Why does RAG cite the right document but return the wrong row?

Document-level recall can pass while the answer-bearing row never enters the context window. That is why generic retrieval metrics under-detect table failures: they score the page, not the row. Patriwala’s point that embeddings are not relational databases applies here: once row relationships disappear, the right PDF can still supply the wrong entity’s value. VDF AI states the user-visible form as an answer that comes back wrong, right for the wrong period, or confidently attached to the wrong row.

When the gold row text is present and the numeric string inside it still disagrees with the source cell, the failure has moved from row selection into digit corruption. That check belongs to OCR and parser fidelity, not to another top-k widen, because ranking cannot restore a digit the extraction artifact already changed.

Did OCR or parsing change the digits in the cell?

If the chunk’s numeric string does not match the source cell, retrieval never had a true fact to return. No amount of prompt tuning recovers a digit the parser already corrupted. Towards Data Science treats the string-not-table path as the root of later chunk-size tuning that cannot recover precision: every lever the team measures sits on top of the parser’s output, and a flattened table on page 47 produces noise no chunker can repair.

When the digits match the source cell and the answer still misstates the quantity, look next at the unit, scale, period, or footnote that never entered the chunk beside the number.

Is the number right but the unit or scale wrong?

RAG often returns a magnitude that matches a cell while dropping the unit, scale note, period, or footnote that changes its meaning: in millions, unaudited, currency, per-share versus total, monthly versus quarterly. That is a distinct failure from a wrong digit. The characters look plausible; the quantity is not. note.com names dropping units and periods as the most dangerous remaining pitfall after teams believe they have handled tables, because a plausible bare number is unusable in practice.

Even with units carried on every row, continuation pages and merged parent labels can still orphan the fact that answered the question. A second-page row that lost its carried header fails the same way a bare number fails: the magnitude looks right while the label that made it usable never entered the prompt.

Did a multi-page or merged-cell table lose the fact?

When a table continues across pages or a label spans merged cells, continuation rows often arrive without headers and child rows lose the parent label. Humans read the carry-forward; page-wise extractors usually do not. Patriwala lists merged cells and parent-child relationships among the structures embeddings cannot reconstruct after flatten. VDF AI’s structural fix explicitly recovers merged cells and multi-level headers before serialising each row with its own context.

Some wrong numbers never came from a single cell at all. Before spending another week on chunking, ask whether the question was a retrieval question: aggregates, trends, and cross-table totals often need a query tool rather than another embedding pass over shredded rows.

When is a table question not a retrieval problem at all?

A point lookup about a specific figure in a specific document is a retrieval question. An aggregate, trend, or cross-period comparison often is not, because the answer exists in no single chunk and must be computed. VDF AI’s second cause states that design answer as routing: point lookups go to the document index; aggregates go to a query tool over the system of record. On the BIRD text-to-SQL benchmark (12,751 question-SQL pairs across 95 databases in 37 professional domains), VDF cites leading published systems in the low eighties for execution accuracy against a measured human benchmark of 92.96 percent, which is why a generated query should be treated as a draft shown beside returned rows.

Patriwala draws the same line for row and column lookups versus narrative summarisation: questions such as the value in row 14, column 3 need table retrieval or structured lookup, not the same vector index that serves discussion-section summaries. Zou et al. (2025) build MultiTableQA around answers distributed across multiple tables, with single-hop and multi-hop table QA plus table-based fact verification, which is another signal that multi-table arithmetic is a different problem class than one-cell retrieval. Full dual-store and TableRAG architecture depth belongs with structure-preserving extract. For a failing point lookup that should have been retrieval, the next move is to run the five checks as one cause-namer.

How do you run the five table checks on one failing query?

Rather than reason about the five causes one at a time, run them against one labelled failure. You need the failing query, the gold coordinates (document, table, row, column header, value, unit), the retrieved chunk texts, and the source cell string. The function below prints which cause fired. It depends only on string checks and your top-k row ids; no new model is required. VDF AI’s evaluate-tables-separately rule matches the same instrumentation: score whether the right table was retrieved, whether the right row and column were identified, and whether the number was reported with its correct unit and period, because a pipeline that gets the first two right and the third wrong is the dangerous case.

Store row_id, column names, header-carried flags, and extraction artifact version as metadata on the same Weaviate objects you retrieve in production, or on whatever store serves the table chunks, so the printed cause matches live behaviour. Keep the failing query in a labelled table slice so the checks run in CI under RAG regression testing.

import re

def _norm_num(s: str) -> str:
    """Keep digits and one decimal point for digit_match."""
    s = s.replace(",", "")
    m = re.search(r"-?\d+(?:\.\d+)?", s)
    return m.group(0) if m else ""

def diagnose_table_number(chunks, *, gold, top_row_ids, k=5):
    """Print which table-number failure fired for one labelled query.

    chunks       -> list[str]  retrieved chunk texts (top-k order)
    gold         -> dict with keys:
                    row_id, col_header, value, unit, page, merged_parent
                    source_cell (raw string from the PDF/Excel cell)
    top_row_ids  -> list[str]  row ids aligned with chunks (or [] if unknown)
    """
    text = "\n".join(chunks)
    header = gold["col_header"].lower()
    unit = (gold.get("unit") or "").lower()
    source = gold["source_cell"]

    # Cause 1: header loss: value region without its label
    if header and header not in text.lower():
        print("CAUSE 1 header loss -> carry column headers into every row chunk")

    # Cause 2: wrong row: document may be present; answer row is not
    if top_row_ids and gold["row_id"] not in top_row_ids[:k]:
        print(f"CAUSE 2 wrong row (row_hit@{k}=0) -> structure-aware row chunks")

    # Cause 3: digit / OCR corruption
    src_n = _norm_num(source)
    if src_n and src_n not in text.replace(",", ""):
        print("CAUSE 3 digit/OCR mismatch -> fix extraction before embeddings")

    # Cause 4: unit / scale missing beside the number
    if unit and unit not in text.lower():
        print(f"CAUSE 4 unit/scale missing ({gold['unit']!r}) -> serialize unit with value")

    # Cause 5: continuation / merge loss
    needs_cont = int(gold.get("page") or 1) > 1 or bool(gold.get("merged_parent"))
    if needs_cont and header and header not in text.lower():
        print("CAUSE 5 continuation/merge loss -> merge multi-page tables before chunking")

    print("done: delete ruled-out causes; fix the first that remains")

Each print line names a row of the diagnosis table. After a cause is confirmed, change only that lever, then re-run the golden pair so the same wrong figure cannot return silently.

A wrong table number is almost never a mysterious model failure. It is one of five structure faults (lost headers, wrong row, corrupted digits, missing units, or broken continuation) or a question that should have been routed to computation rather than retrieval. Inspect the retrieved table chunk and the extraction artifact first, run the five discriminating checks on a labelled gold cell, keep that failure in a table-specific eval slice, and send mechanism depth to ingestion and table chunking only after the cause is named.