At EMNLP I'm the most excited about this work which mixes theory with actual practicality. Given N models, how do you efficiently find which ones are the top-3? Evaluating everything is VERY expensive. aclanthology.org/2026.finding...
aclanthology.org
In the cESA annotation protocol, the annotators see multiple outputs at the same time, which they judge with error spans and absolute scores. This is faster and more objective than doing it one by one. It's implemented in Pearmut and used in WMT26. arxiv.org/abs/2607.26640 WMT
Contrastive ESA: Human Evaluation of Multiple Translations at Once arxiv.org
AI Weekly's analysis
→
- cESA shows annotators several translations of the same source at once, has them mark major and minor error spans, then score each 0-100%.
- The method was validated on an English-to-Japanese comparison of 12 models, with the authors reporting lower annotation time and noise than pointwise evaluation.
- Unlike contrastive ranking, cESA yields absolute quality judgments that support non-parametric model rankings without post-hoc statistical corrections.
Read full analysis →
Dynamic annotation allocation is a way to focus on better-performing candidates as the annotations progress. Heavily inspired by bandits and we prove the surprisingly A-optimality of a simple algorithm. It's implemented in Pearmut and used at scale in WMT26. arxiv.org/abs/2608…
Dynamically Allocating Evaluation Effort for Model Ranking arxiv.org
Pearmut is a tool to quickly set up reproducible human annotations. It mostly replaces Appraise at WMT, but already found many uses outside given its flexibility, incl. speech annotation and general text annotation. A small human eval can elevate most papers. arxiv.org/abs/260…
Pearmut: Human Evaluation of Translation Made Trivial arxiv.org
The Last Translation Benchmark is still open (10 accepted submissions in any language/dialect for coauthorship 🤩). last-translation-benchmark.vilda.net Also submit your model & try to beat LTB. We added all the latest models: North Small Translate (@cohere.com), Mistral Large …
Last Translation Benchmark last-translation-benchmark.vilda.net
View on Bluesky ·
♥ 13
↻ 3
↩ 0
·
3 from the directory shared this ·
36d ago
AI is one such tool. Skeptics say that modern science will devolve into mathematics-as-chess, or mathematics-as-poetry [4], but I'm skeptical of these claims because science was never about generating papers. [4] The Future of Human Mathematics. Jacob Hilton. 2026 www.lesswron…
Jacob_Hilton's Shortform — LessWrong lesswrong.com
