Nature paper: AI harms in healthcare bypass treating physicians

A new Nature Health paper went through 31 US legal cases involving medical AI and found that the tools ending up in court were mostly not the ones sitting in front of treating physicians. They were the ones deployed by insurers, state agencies, and care facilities on the other side of a patient's care.

Gennie Mansi and Mark Riedl of Georgia Tech write that 'patients' medical care relies on a complex web of stakeholders,' and that 'many AI tools deployed across their healthcare delivery negatively impact their care.' The named AI systems in the case pool include nH Predict, the Oregon DHS Safety at Screening Tool, and the Babylon Health Triage System.

The framing the authors push back on is the one that treats the physician as the last line of defense. Their reading of the record is that 'patients have had no option but to seek legal recourse for harms,' because the harms are landing in coverage denials, state screening decisions, and triage, places physicians do not sit.

Their two proposed responses are practical rather than sweeping: 'changing liability structures to reflect the role of many stakeholders,' and designing tools that let patients' lawyers actually build cases. The analysis draws on three public databases, the Database of AI Litigation, the O'Neill Institute Health Litigation Tracker, and the AIAAIC repository, reviewed as of February 2025.