Paper: clinical AI benchmarks don't test documentation work

Documentation and administration made up 36.2% of 127,833 queries sent by 6,342 physicians, advanced practice providers and nurses to an institutional LLM assistant during an eight-month roll-out. Diagnosis made up 3.7%.

The median of 58 public benchmarks used to judge whether these systems are ready for clinical deployment contained no documentation requests at all.

That mismatch is the finding of a new paper on arXiv that built a framework called RCQ-Map to characterize each query by task, intent, answerability, missing information and potential harm, then applied it to the real queries across 35 specialties and to 58 public benchmarks drawn from "major evaluation suites and frontier model reports."

Knowledge retrieval was the second-biggest category of real use at 28.9%. Together with documentation it covered "nearly two-thirds of use." More than a third of queries could not be answered well as posed.

"Benchmark scores therefore say little about how clinical AI performs on most of the work it is actually given," the authors write, "and evaluation should be matched to real clinical use." Benchmarks in suites explicitly designed to resemble clinical practice were "individually no closer to real use than those used in frontier model reports."