Nature Medicine: benchmarks can't earn medical AI's trust

Trust in clinical AI "cannot be benchmarked into existence," Mike Schaekermann and colleagues argue in Nature Medicine; it "must be earned through rigorous prospective studies in real-world clinical settings," where "the hardest lessons often concern the humans and systems around the AI, not the technology itself."

The argument arrives with a prospective feasibility trial of AMIE, a conversational diagnostic AI, run at Beth Israel Deaconess Medical Center's Healthcare Associates clinic in Boston from April 2025 to November 2025. One hundred adult patients interacted with AMIE by text-chat up to five days before their appointments, and human safety supervisors monitored every session in real time. According to the trial, "zero safety stops were required by the human AI supervisors."

The numbers are respectable, though the setting is one clinic. AMIE's differential diagnosis included the final diagnosis "per chart review 8 weeks post-encounter, in 90% of cases, with 75% top-3 accuracy," and reviewers found "similar overall DDx and Mx plan quality between AMIE and PCPs" without significant differences on appropriateness or safety of management plans. Primary-care providers still outperformed AMIE on treatment practicality (p=0.003) and cost-effectiveness (p=0.004). Patient attitudes "shifted more positive after interacting with AMIE and remained at an elevated level after seeing the provider."

Two of the AI experts we follow posted the commentary the day it appeared.