Last Translation Benchmark ships MT hard cases with fail rules

The Last Translation Benchmark, posted on arxiv, is a collection of examples "that break leading machine translation models," each paired with "handcrafted verification rules describing concrete failure cases." The dataset is designed to stay live and accept ongoing contributions. LTBv1, the current release, covers submissions accepted before September 1st 2026 and spans texts, images, audio and videos.

The paper is blunt about why it exists. The authors write that "standard benchmarks for machine translation are approaching saturation," and that automatic metrics are "unreliable, vulnerable to reward-hacking, and provide unactionable assessments." Even gold human evaluation, the paper says, "often lacks reproducibility, objectivity, and scalability."

Instead of assigning a score, each item defines what a failing translation would look like on that specific example, so a model either trips a rule or it does not.

Led by Vilém Zouhar and Niyati Bafna, the collaboration lists hundreds of contributing authors. Two researchers on our radar had shared the arxiv link within a day of its posting.