AMIE team: benchmarks can't earn clinical AI trust

"Trust in clinical artificial intelligence (AI) cannot be benchmarked into existence." That is the opening line of a commentary in Nature Medicine from the Google team behind AMIE, its conversational diagnostic system.

The piece is coauthored by Mike Schaekermann, Anil Palepu and colleagues at Google Research and Google DeepMind, together with Adam Rodman of Beth Israel Deaconess Medical Center and Harvard Medical School, Ami Parekh of Included Health, and Ethan Goh of Stanford. Their argument is that clinical trust "must be earned through rigorous prospective studies in real-world clinical settings, where the hardest lessons often concern the humans and systems around the AI, not the technology itself."

The commentary rides alongside a feasibility trial at Beth Israel's ambulatory primary care clinic. One hundred adult patients interacted with AMIE by text-chat before their appointments; ninety-eight kept the scheduled visit. AMIE's differential included the final diagnosis in 90% of cases, top-3 accuracy was 75%, and human safety supervisors called zero stops. A blinded panel judged AMIE and primary care providers to have similar overall differential and management plan quality, though the PCPs came out ahead on practicality and cost-effectiveness of the management plans.

The judgment the authors are pressing on is not about model quality. It is that leaderboards, however good, cannot substitute for watching what happens when patients and clinicians actually use the thing. Two of the researchers in our Who's Who directory shared it within the first week.