Speech Event Detector adds 54 sound tags to Whisper transcripts

A 5.8-million-parameter detector tags 54 kinds of non-verbal sound alongside the words Whisper already picks out: 20 speaker vocalizations such as coughs, breaths and laughter, and 34 background sounds including horns, music and barking. The convolutional-Transformer, described on arXiv by Ayush Kumar Bar and Promila Ghosh, emits probabilities every 40 milliseconds and converts them into timestamped events, with code and weights released under a non-commercial licence.

The paper opens on the gap plainly: "Automatic speech recognition produces words but drops the non-verbal content of a recording: the breaths, coughs, sighs and laughter of the speaker, and the horns, music or barking around them."

Training drew on seven public datasets with incompatible labelling formats, unified through rule-based mapping and CTC forced alignment, and combined about 168 hours of annotated real recordings with synthetic mixtures and background-noise augmentation.

Accuracy is modest. On held-out test recordings the sensitive operating profile hits 49% recall at 0.30 precision; the balanced profile sits at 0.37 recall and 0.36 precision, with macro event F1 scores of 0.32-0.33. The detector runs about 39 times faster than real time on eight CPU threads.