nature.com web signal

ML model tops Copenhagen call-takers on stroke recognition

TL;DR

  • On 2021 Copenhagen test calls, a machine learning pipeline hit 63.0% sensitivity for stroke versus 52.7% for human call-takers, with p < 0.0001.
  • Positive predictive value rose from 17.1% (call-takers) to 24.9% (ML), both measured on the 1-1-2 emergency and MH-1813 medical helpline lines.
  • The paper reports better model performance on patients aged 65 and older than 18–64, and higher PPV and F1 for men than for women.

A machine learning pipeline recognised stroke in Copenhagen medical helpline calls more often than human call-takers did, according to a paper in npj Digital Medicine. On 2021 test calls, call-takers achieved a "sensitivity of 52.7% (95% confidence interval 49.2–56.4%) with a positive predictive value (PPV) of 17.1%". The model achieved "a sensitivity of 63.0% (62.0–64.1%) and a PPV of 24.9% (24.3–25.5%)", with p < 0.0001.

The pipeline is two stages. An automatic speech recognition model transcribes the call, then a text classifier labels it stroke or non-stroke. Training used "calls from 1 January 2015 to 31 December 2020 in Copenhagen" across the 1-1-2 emergency line and the MH-1813 medical helpline, run by Copenhagen Emergency Medical Services. Lead author Jonathan Wenstrup is at the Department of Neurology, Copenhagen University Hospital, Herlev and Gentofte; several co-authors list Corti, a Danish health-AI firm, alongside the Technical University of Denmark and Aalborg University.

The authors are cautious about the framing. "This study does not imply that a machine-learning model can replace medical call-takers," they write.

Performance is not uniform. The paper reports better results on patients aged 65 and older than on those 18–64, and higher PPV and F1 for men than for women. The paper also notes the gap over human call-takers is larger than in a prior cardiac arrest ML study by the same line of work.

Shared on Bluesky by 1 AI expert