Nature study: AI healthcare harms rarely start with doctors

A Georgia Tech team reviewed 31 US legal cases involving AI tools in healthcare and found that, in most of them, the algorithm doing the damage was not the one the treating physician was running. It belonged to the insurer, the state health department, or the care facility.

Gennie Mansi and Mark Riedl make the argument in Nature Health. "Patients' medical care relies on a complex web of stakeholders—physicians, state health departments, health insurers and care facilities, among others," they write, "and many AI tools deployed across their healthcare delivery negatively impact their care." Named in the paper are the nH Predict system, the Oregon DHS Safety at Screening Tool, and the Babylon Health triage system.

Their recommendations are two. The paper calls for "changing liability structures to reflect the role of many stakeholders in shaping how AI tools impact patient care," and for "designing tools that can help advocates, such as patients' lawyers, provide critical legal expertise and practically support recourse for patients." That second one reframes a lot of the explainable-AI research agenda: build the tools for the plaintiffs' bar, not the attending physician.

The paper does not break out which cases ended in which rulings, or attach a dollar value to any settlement. Mansi and Riedl work upstream of that. Patients, they write, "have had no option but to seek legal recourse for harms."