How would you design an Agentic AI system to assist doctors in a hospital setting?

What you need to know

Clarify first

• Users: physicians, nurses, pharmacists or admin staff have different needs and permissions.
• Decisions: supported (drafts and suggestions) versus made (orders). The agent makes none.
• Integration: the EHR, usually through FHIR APIs; lab, imaging and pharmacy systems.

The flow

1. Authenticate as the clinician; inherit their access, nothing more.
2. Assemble context — problems, medications, allergies, recent labs and vitals, prior notes, each with a date.
3. Retrieve guidance — hospital protocols and approved guidelines.
4. Produce a bounded artefact — note draft, admission summary, interaction check, or suggested orders for review.
5. Clinician reviews, edits, signs.
6. Write back to the EHR, marked as AI-drafted, with the edits logged.

Controls

• Every clinical statement cites a record element or guideline, with an as-of date.
• No autonomous orders and no autonomous patient messaging.
• Dosing, interactions and risk scores come from deterministic tools and databases.
• The agent abstains and says what is missing rather than filling gaps.
• Safety-critical fields, such as allergies and active medications, are pinned from structured data and can never be summarised away.

Failure modes

Hallucinated findings, old labs shown as current, context overflow dropping the allergy list, and automation bias: a busy clinician signing a plausible but wrong draft. Mitigations: visual marking of generated versus sourced text, "data as of" labels, schema-validated output, and review sampling.

Metrics and first release

Documentation time saved, edit distance on drafts, clinician acceptance rate, and every safety event reviewed individually. Ship ambient note drafting and chart summaries first: high burden, reversible, and the clinician is always in the loop.

A real-life example

A hospital's internal medicine ward pilots discharge summary drafting with 20 doctors.

• The agent assembles the admission reason, daily progress notes, procedures, final labs, and the discharge medication list from the EHR.
• It drafts the summary with each sentence linked to a source note or result. Medications and allergies are copied from structured fields, not generated.
• A deterministic interaction checker runs on the discharge medications and flags one combination for the doctor.
• The doctor edits and signs. Median edit: about 15% of the text.

Writing time per summary fell from about 25 minutes to about 8. In the first month, reviewers found two drafts that stated a lab value from an earlier day as the latest. The fix added a "latest value and date" field from structured data, and the issue did not recur.

Follow-up questions to expect

• "How do you handle missing data?" — Show it explicitly ("no creatinine in the last 48 hours") instead of guessing. Abstaining is a feature.
• "How do you fight automation bias?" — Highlight uncertain or unsourced items, make sources one click away, and track how often drafts are signed with no edits in suspiciously short time.
• "Where does the model run?" — Where patient-data rules allow: a provider with the right agreements and data residency, or a hospital-hosted model.