What happens if retrieved documents contradict each other?

What you need to know

There are two different kinds of contradiction, with different fixes.

1. Fix it in the data first

• Store effective_date, version and status (active or superseded) on every chunk.
• When a new version arrives, remove or mark the old chunks at ingest. Do not rely on ranking to hide them; an old policy can be a better semantic match than the new one.
• Store authority (for example official, seller, community) so you can filter or order by it.

2. Tell the model what to do

Each document has an effective_date and an authority.If documents disagree:- Prefer the most recent effective_date among official documents.- Say that an older or unofficial source disagrees, and cite both.- If you cannot tell which applies, say so instead of choosing.

Surfacing a conflict is a better answer than hiding it. "The 2026 policy says 12 WFH days a month [1]; an older 2024 FAQ says 8 [3]" gives the user what they need.

3. Detect it automatically

For high-risk topics, run a cheap check over the retrieved set: ask a small model or an NLI model (natural language inference, which labels a pair of sentences as agreeing, contradicting or unrelated) whether any two chunks contradict each other on the question. If yes, log it and, depending on risk, show both or escalate.

Conflict logs are also a to-do list for the content team: each one points at a document that should be updated or retired.

A real-life example

An HR policy assistant is asked, "How many work-from-home days do I get?" Retrieval returns the April 2026 hybrid-work policy (12 days a month) and a 2024 FAQ page (8 days a month) that nobody deleted. The model answers "8 days", because the FAQ is written in simpler question-and-answer style and matches the question better.

The engineer makes two changes. Ingest now reads the status field from the HR document system and skips anything marked superseded, which removes 140 old chunks. The prompt now includes effective_date in each <doc> tag and the rule above. In the following month, the conflict detector still fires on 23 questions, all caused by two team wiki pages that repeat old rules. HR updates those pages, and the conflicts stop.

Follow-up questions to expect

• "Why not just sort by date?" — Date helps with time conflicts, but a newer blog post should not beat an older official policy. You need both date and authority.
• "What if both documents are current?" — Then it may not be a conflict: rules can differ by country, grade or product. Add those fields to metadata and filter by the user's attributes.
• "How do you test this?" — Put known conflict pairs in the evaluation set and check that answers either pick the right source or clearly mention both.