Speech BCI Paper Debuts OVMI, Flags Inflated Accuracy Scores

A new arXiv paper by Dulhan Jayalath, Benjamin Ballyk and Oiwi Parker Jones argues that the standard scoreboard for speech brain-computer interfaces is misleading, and offers a replacement.

The paper introduces open-vocabulary mutual information, or OVMI, an information-theoretic quantity that "measures the information conveyed by a decoder relative to a reference distribution over the words a user may wish to communicate." The point of the exercise is comparability: systems today use different datasets, recording methods, and vocabularies, and, as the authors write, "their reported scores are rarely comparable."

The critique of current practice is blunt. Accuracy, word error rate and similar metrics "computed only over the words a system supports can overstate how much of a user's intended speech the system can communicate." Score high on a small curated vocabulary and the number looks strong; the share of what the user actually wants to say may not.

OVMI is also pitched as a design tool, not only a referee. The authors report that "selecting a vocabulary to maximise OVMI yields up to 16.3% relative improvement in accuracy across three speech domains," and frame the metric as "a principled way to compare heterogeneous systems, improve vocabulary design, and measure progress in the field."

The abstract does not name which existing speech BCI systems were re-scored under OVMI, nor which reference word distribution the comparisons used. Those choices will decide how much any given lab's headline number moves.