How do you design prompts to prevent hallucinations in RAG?

What you need to know

Models hallucinate in RAG for three main reasons: the answer is not in the context, the context is confusing, or the model prefers what it learned in training. The prompt can help with the second and third. Nothing in the prompt can help with the first, which is the most common.

The prompt levers

• Scope. "Answer only from the documents below." Simple, but it sets the task.
• A named escape hatch. "If the documents do not contain the answer, reply exactly: I could not find that in the guidelines." A model with no allowed way to say "I don't know" will produce something.
• Citations per claim. "Put the document number after every sentence, like [2]." A sentence that must point to a source is harder to invent, and your code can check it.
• Quote first, then answer. Ask the model to copy the exact supporting sentences, then answer from those. This keeps it anchored, and Anthropic's long-context guidance suggests exactly this for long documents.
• Clear separation. Rules in the system message, documents in tags, question last.

A compact version:

You answer staff questions using the hospital's clinical guidelines.Rules:1. Use only the text inside <documents>.2. First list the exact sentences that support your answer, with their [id].3. Then answer in at most 4 sentences. Every sentence ends with a citation like [2].4. If the documents do not answer the question, reply exactly:   "I could not find that in the guidelines."

The levers outside the prompt

• Relevance floor. If the best reranker score is below a threshold you tuned, skip the model call and return the fallback. It costs nothing and removes the most common cause.
• Citation check. After generation, confirm every cited id was actually retrieved and every sentence has one (code in the lesson on limiting answers to context).
• Faithfulness scoring. On a sample of traffic, have a judge model check each claim against the cited chunk.
• Low temperature, where the model allows it, so the answer does not drift into creative text.

A real-life example

A hospital's clinical-guideline search tool is asked, "What is the paediatric dose of drug X for a 12 kg child?" The guideline store has the adult protocol only. The first prompt version said "use the context", with no escape hatch. The model scaled the adult dose by weight and gave a confident number, which is exactly the kind of answer that must never appear.

The team makes three changes. They add the exact fallback sentence. They add a relevance floor: if no chunk mentions both the drug and "paediatric" or "child" with a reranker score above their threshold, the tool replies "I could not find a paediatric protocol for this drug; please check the formulary or ask pharmacy." And they add a rule in code that any number in the answer must appear in a cited chunk. On 150 test questions written by pharmacists, including 30 whose answers are deliberately missing from the store, invented answers to the missing-answer questions go from common to rare, and none include an invented dose.

Follow-up questions to expect

• "Does 'only use the context' stop the model using its own knowledge?" — Not completely. It lowers it. That is why you measure faithfulness instead of trusting the instruction.
• "Won't the escape hatch make it refuse too much?" — It can. Track the unhelpful-refusal rate on questions that do have answers, and tune the prompt and the relevance floor against both numbers.
• "Is a bigger model less likely to hallucinate?" — Often it follows grounding instructions better, but it can also fill gaps more fluently. Measure it on your data.