Copenhagen ML spots stroke on helpline calls better than humans

On non-emergency medical helpline calls in Copenhagen, a machine learning pipeline recognized stroke with 63.0% sensitivity and 24.9% positive predictive value. Human call-takers on the same 2021 test set managed 52.7% and 17.1%. Both gaps are statistically significant at p<0.0001, according to a retrospective study in npj Digital Medicine.

Stroke appeared in 0.250% of those calls.

The system stacks two models. First, automatic speech recognition, built as a neural network with convolutional and bidirectional LSTM layers and trained on 3,811 manually transcribed emergency 1-1-2 calls totaling 173 hours, turns the call into text. Then an ensemble of five multi-layer perceptrons, fed bag-of-words n-grams, classifies the transcript. Occlusion analysis flagged the words doing the predictive work: 'double vision', 'difficulties speaking' and body-part laterality ('left', 'right', 'side').

Training data ran 2015 through 2020; the test set was every MH-1813 call from 2021. Copenhagen Emergency Medical Services covers 1.9 million residents, and the authors describe the work as 'the largest collection of audio calls from patients with stroke to date.'

Performance was uneven across patients. The paper reports significantly higher PPV and F1-score in men than women, and significantly better sensitivity, PPV and F1-score in patients aged 65 and over versus those 18-64, both at p<0.0001. The authors note no data was collected on 'bias related to race, ethnicity, language, accent, or dialects,' and the speech model is Danish only.