Kyutai releases MuScriptor for multi-instrument transcription

Automatic music transcription has been stuck in an awkward spot for years. Models that could handle a solo piano cleanly would fall apart the moment you fed them a band recording with drums, bass, guitar and vocals stacked together. Kyutai and Mirelo just published a paper on arXiv describing MuScriptor, an open decoder-only Transformer they claim closes that gap.

The setup is straightforward. The model reads a mel-spectrogram of a short audio segment and autoregressively predicts MIDI-like tokens for pitch, timing and instrument. The training recipe is where the interesting choices are. Rather than lean entirely on synthetic MIDI, which is the trap that made prior open models unusable on real mixes, the team paired 1.45 million synthetic MIDI files for pre-training with roughly 170,000 real recordings spanning over 11,000 hours of aligned note annotations, then applied reinforcement learning post-training. Three checkpoints are up on Hugging Face: small at 103M parameters, medium at 307M, and a large variant at 1.4B.

The reported gains over the prior open baseline are big. MarkTechPost's write-up notes MuScriptor's large variant hits a Multi F1 score of 48.2 on test data compared to 21.9 for the YourMT3+ baseline, and lifts onset F1 from 32.5 to 60.4 and frame F1 from 45.5 to 73.3. Take the specifics as reported, not settled. Benchmark deltas on a self-authored paper often shrink under independent evaluation.

The honest caveats. Weights ship under CC BY-NC 4.0, so anyone hoping to drop MuScriptor into a paid product will need a different arrangement, since the inference code is MIT but the model itself is not commercially usable off the shelf. The paper also does not disclose which 170,000 songs make up the real-audio training set, which matters given the ongoing lawsuits against music-AI companies over exactly that question. Latency, memory footprint and how the model behaves on vocals versus pitched instruments are not spelled out in the reporting either.

Even with those asterisks, the direction is what to watch. A free, high-accuracy stem-to-MIDI converter is the missing plumbing for a lot of downstream tools: music-education apps that generate lessons from any recording, DAW plug-ins that give you an editable MIDI track from a live take, sample librarians labelling huge audio archives. Whoever bolts a commercial license or a hosted API onto something equivalent captures a wedge of that market first.