Course Content
Fine-Tuning LLMs
6 sections · 52 lessons
How do you fine-tune for legal, medical, or finance domains while ensuring accuracy and risk control?
What you need to know
Match each risk to its control
| Failure | Example | Main control |
|---|---|---|
| Wrong or outdated fact | Cites a repealed section | RAG over a dated, versioned source store |
| Invented or wrong citation | Quotes a paragraph that says something else | Check at inference that the quoted text exists and supports the claim |
| Overconfident advice | Gives a dosage when the report is ambiguous | Train abstention; human review for high-stakes outputs |
| Privacy leak | Repeats a patient's name from training data | Scrub personal data before training; output filters |
| Unauditable decision | "Why did it say that?" | Log inputs, retrieved sources, model and adapter version |
What fine-tuning is good for here
- Form: the structure lawyers or doctors expect — findings first, then impression; issue, rule, application, conclusion.
- Terminology and tone: domain phrasing, careful hedging, mandatory disclaimers.
- Abstention: include training examples where the right answer is "the provided documents do not cover this; please consult a specialist". Without them, the model fills every gap confidently.
Data rules
- Expert-written or expert-reviewed outputs only; record who reviewed each example.
- Remove or mask personal data before it reaches a GPU. In India, personal data processing falls under the DPDP Act 2023, and health and financial records carry extra sector rules.
- Check the licence and consent for every source.
Evaluation
- Factual accuracy against an expert answer key.
- Citation validity — does the cited passage exist, and does it support the claim?
- Critical-error rate — errors weighted by harm; one missed cancer finding is not equal to one typo.
- Appropriate-refusal rate — does it decline when it should, and only then?
A real-life example
On 1 July 2024, India's new criminal codes came into force: the Bharatiya Nyaya Sanhita replaced the Indian Penal Code. Cheating, long known as "IPC section 420", is now covered by section 318 of the BNS.
Imagine a legal assistant that was fine-tuned in 2023 with statutes in its training answers. It will keep citing IPC sections with confidence, and the only fix is retraining. Now imagine the design above: statutes live in a retrieval store with effective dates, so the firm updates the store and the assistant cites BNS the same day. The fine-tune taught only how to write a legal note — cite the section, quote the text, flag which code applies based on the offence date — and that behaviour stays correct when the law changes. Associates still review every note before it reaches a client.
Follow-up questions to expect
- "Would you ever put domain facts in the weights?" — Stable background knowledge such as terminology and common concepts, yes, through continued pretraining. Anything that is cited, changes or must be audited, no.
- "How do you stop invented citations?" — Require the model to quote the exact passage, then check in code that the quote exists in the retrieved document; reject or regenerate if it does not.
- "Who signs off before launch?" — Domain experts on the evaluation results, plus legal or compliance for data use; the human-review gate stays after launch.