By sector

Healthcare: the risk of clinical hallucination in AI assistants

July 30, 2026 6 min read
Kamel Malek Director of SetRecord.ai

Between two consultations, a physician has ninety seconds and a question a decision-support tool wasn't built to answer quickly: an unusual drug combination, a dosage adjustment for a patient with reduced kidney function. Increasingly, she asks a general-purpose AI assistant instead of paging a pharmacist. Most of the time it answers correctly. The exceptions are the problem.

The question asked between two patients

Clinical decision-support systems exist precisely because medical knowledge changes faster than any single clinician can track, and because getting it wrong carries direct human cost. They are built, validated and updated under regulatory scrutiny most consumer AI assistants never face.

Yet the fastest tool in the room is now often a general-purpose assistant open in a browser tab — not because clinicians trust it more, but because it is instantly available and answers in plain language. The workflow it replaces, informally, is a corridor conversation with a colleague or a call to the pharmacy. The workflow it does not replace, and was never validated against, is the formulary itself.

The same dynamic shows up across specialties with very different stakes. A general practitioner checking a paediatric dosing range, an emergency physician confirming an antidote protocol at 3 a.m., a nurse verifying a discharge instruction — each represents a moment where speed is genuinely valuable, and where the underlying tool was never certified for the specific question actually being asked of it.

How a guideline gets misquoted

Clinical hallucinations share a structural cause with financial ones: a model reconstructs a plausible-sounding answer from fragments, rather than retrieving a single authoritative, dated source. In a clinical context, the recurring patterns we observe are:

  • Superseded guidance — a dosage or contraindication that was accurate under a prior version of a guideline, stated as current.
  • Population conflation — a finding from a trial in one patient population (age group, comorbidity profile) generalised as if it applied universally.
  • Invented interactions — a drug-drug interaction warning that sounds cautious and plausible but does not appear in either drug's approved label.
  • Confidence without a caveat — an answer stated in the same assertive register whether the underlying evidence is a large randomised trial or a single case report.

A representative case we see repeatedly: asked about a common anticoagulant interaction, an assistant correctly names the interacting drug class but states the recommended action — dose reduction rather than avoidance — based on an older edition of the guideline that a subsequent revision reversed. The answer sounds precise because it names the right drugs. It is wrong because it stopped reading the evidence one revision too early.

None of this stems from the model attempting to deceive. It is the predictable output of a system trained to sound authoritative, applied to a field where the difference between two similar-sounding answers can be the difference between a safe dose and an unsafe one.

Why clinical claims need a higher evidentiary bar

Not all sources are equal, and medicine has spent decades building formal hierarchies to say so: a peer-reviewed randomised trial outranks an observational study, which outranks a case report, which outranks an unreviewed forum post. A model that treats a drug-information forum and a regulator-approved product label as interchangeable text to pattern-match against has, in effect, discarded that hierarchy.

This is precisely the gap our own methodology was built to make visible outside healthcare too: not whether a claim sounds confident, but whether it traces back to a source with the evidentiary weight the claim requires.

This is not an argument against AI assistance in medicine — plenty of clinical-decision tools are held to exactly this evidentiary standard and perform well within it. It is an argument for knowing which category of tool is actually in use at a given moment, and for not letting the fluency of a general-purpose assistant borrow credibility that only a validated one has earned.

Don't know what AI says about you today?

A SetRecord audit queries supported AI models about your organisation and measures the gap with verifiable facts — within five business days.

Request an audit

The patient-safety stakes

A factual error about a company's revenue is a reputational problem. A factual error about a drug interaction, relayed with confidence to a clinician who trusted the tool to save time rather than replace judgment, is a different category of risk entirely. It doesn't need to happen often to matter — a single incorrect answer, acted on once, is already a patient-safety event.

Health institutions that have not yet mapped where clinicians already turn to general AI assistants are not avoiding the exposure; they are simply unable to measure it. The tools are in use whether or not they are sanctioned, which makes silence the riskiest posture of all.

Insurers, hospital counsel and clinical-safety officers increasingly ask the same question in different words: if an incorrect AI-sourced answer contributed to a care decision, who is accountable, and what evidence exists that the institution had any visibility into the exchange at all? An organisation with no monitoring in place has no answer to either question — which is itself the exposure most worth addressing first.

What a credible clinical answer looks like

A trustworthy AI answer in a clinical context should be traceable to a named, dated, versioned source — a current product label, a specific published guideline with its revision date, a systematic review rather than an individual study. It should distinguish between an approved indication and an off-label use. And, critically, it should be able to say "the evidence for this is mixed" or "I don't have a current answer" rather than defaulting to fluent certainty.

This standard is not exotic — it mirrors what a well-run hospital formulary committee already expects from any new source before it earns a place in clinical workflow. Applying the same bar to an AI assistant's answers is less a new requirement than an extension of existing practice to a channel that arrived faster than governance could formally adopt it.

A verification protocol for health institutions

Institutions do not need to ban general AI assistants to manage this risk — that ship has largely sailed. They need a way to see what the assistants their staff already use are actually saying. A workable protocol:

Map the real questions

Identify the drug interactions, dosage edge cases and guideline questions clinicians are most likely to ask an AI assistant informally, based on your own specialty mix.

Query and compare

Run those questions across the major assistants and compare each answer against your institution's current formulary and adopted guidelines — not against what sounds reasonable.

Track guideline updates

Re-test after every guideline revision. A model that was accurate in one training cycle can become quietly wrong the moment clinical practice moves and its underlying sources do not.

Going further with SetRecord

SetRecord applies this same audit discipline to healthcare organisations: mapping the clinical questions your staff actually ask AI assistants, scoring each answer against your approved formulary and guidelines, and tracing errors back to their likely source. This sits alongside, not in place of, your clinical governance — read our methodology for how we draw that line.

See what AI says about you

A SetRecord audit checks what supported AI models say against your verifiable facts — within five business days.