How do you design an “evidence-first” tool policy (retrieve → cite → answer)?

What you need to know

1. Retrieve — search and write passages with ids (doc-17#p4) to state.
2. Grade — a small model or reranker scores each passage; keep those above a threshold.
3. Rewrite and retry — if fewer than N good passages, rewrite the query and retrieve again; at most two or three rounds.
4. Answer — only from graded passages, citing [doc-17#p4] after each claim; no tools bound.
5. Verify — every cited id exists; for high-stakes answers, a checker confirms each claim is supported by its passage. Failures route back with the unsupported sentence.
6. Abstain — when the budget is used up, say what could not be verified.

1def after_grade(state) -> str:2    if len(state["good_passages"]) >= 3:3        return "answer"4    if state["retrieval_rounds"] < 2:5        return "rewrite_query"6    return "abstain" if not state["good_passages"] else "answer"

Metrics

• Citation coverage — share of claims with a valid citation.
• Faithfulness — share of cited claims actually supported by the passage.
• Retrieval recall — on a labelled set, did the right passage get retrieved?
• Abstention rate — should be above zero; zero means the system is guessing sometimes.

A real-life example

A health insurer's policy assistant answers "Is cataract surgery covered in my plan, and is there a waiting period?" The graph retrieves 8 passages from the policy wording, grades 3 as relevant (the surgery clause, the waiting-period table, the sub-limit schedule), and answers: "Covered after a 24-month waiting period [policy-221#s4.2], with a sub-limit of Rs 40,000 per eye [policy-221#annex-B]." The verify node confirms both ids exist and the numbers appear in the passages. For "Is robotic surgery covered?", only one weak passage is found after two rewrites, and the assistant replies that the policy wording does not clearly say, and offers a call with an advisor. Before evidence-first, a plain RAG bot had answered this with a confident "yes" that the claims team later rejected.

Follow-up questions to expect

• "Isn't verification expensive?" — A cheap model or string checks cover citation existence; run full entailment checks only on high-stakes routes.
• "Why remove the search tool from the answer node?" — So it cannot fetch new, ungraded content or skip the evidence; it can only use what was checked.
• "How is this different from normal RAG?" — Normal RAG retrieves once and hopes. Evidence-first grades, retries with a cap, verifies citations and can refuse.