AMIE team: benchmarks alone can't earn medical AI's trust

Trust in clinical artificial intelligence cannot be benchmarked into existence. That is the opening line of a Nature Medicine commentary from Mike Schaekermann, Anil Palepu, Adam Rodman and colleagues at Google Research, Google DeepMind, Beth Israel Deaconess Medical Center and Included Health, published alongside a prospective feasibility trial of AMIE, the group's conversational diagnostic system.

Their argument is that leaderboards and vignette studies are not enough. Trust, they write, "must be earned through rigorous prospective studies in real-world clinical settings, where the hardest lessons often concern the humans and systems around the AI, not the technology itself."

The evidence they attach is small but concrete. Between April and November 2025, Google's team ran AMIE with 100 adult primary-care patients at Beth Israel Deaconess in Boston. Patients text-chatted with the system up to five days before their appointment while a physician "AI supervisor" watched live over video and screen share, ready to break in per pre-defined safety criteria. Zero safety stops were required. AMIE's differential diagnoses matched the final diagnosis in 90% of cases; top-3 accuracy hit 75%; the single most likely diagnosis was correct 56% of the time. Primary care providers still outperformed AMIE on management-plan practicality and cost-effectiveness.

The sample skewed younger than the clinic's overall panel, and 98 of the 100 enrolled patients attended their scheduled appointment. Two of the AI researchers on our radar shared the piece the day it went up. The commentary is asking the rest of the field to submit itself to the same kind of prospective evidence rather than treat benchmarks as validation.