Copenhagen ML pipeline outperforms call-takers on stroke calls

The machine framework caught stroke more often than the humans did. On 2021 calls to Copenhagen's non-emergency Medical Helpline 1813, call-takers reached a sensitivity of 52.7% and a positive predictive value of 17.1%. An ensemble of text classifiers running over automatically transcribed audio reached 63.0% sensitivity and 24.9% PPV, with the gap significant at p<0.0001.

That is the headline of a retrospective study published in npj Digital Medicine by researchers at Copenhagen Emergency Medical Services and the Danish AI company Corti. They trained on calls from 1 January 2015 to 31 December 2020 and held out 2021 for testing. Stroke was the ground-truth label in 0.250% of 2021 helpline calls, which is why PPV matters as much as sensitivity.

The authors are blunt about what their dataset cannot show. 'We did not have data to estimate potential bias related to race, ethnicity, language, accent, or dialects,' they write. European data rules also ruled out another step: 'it was not possible to manually transcribe MH-1813 calls to train a new speech recognition model,' so the speech component was trained on 1-1-2 emergency calls instead.

One disparity did surface in-sample. The model's PPV was 25.9% on male callers and 22.8% on female.