Adaptive sampling adds +0.67 to XLM-R multilingual scores
TL;DR
- Trainable per-language sampling probabilities beat uniform realignment by +0.67 average on XLM-R and +0.60 on Gemma 2 9B.
- The method targets extremely low-resource languages by upsampling those contributing most to the realignment loss during training.
- Training wraps an inner-outer optimization loop the authors describe as low-overhead, but the abstract publishes no timing numbers.
An adaptive sampling method for multilingual realignment posts modest gains over the standard uniform baseline: +0.67 average with XLM-R, and +0.60 with Gemma 2 9B.
The paper targets extremely low-resource languages, where the authors argue uniform random sampling of parallel sentences is suboptimal under limited batch sizes. Their fix assigns trainable sampling probabilities per language, so weakly-aligned ones get pulled into more batches as training progresses. "Languages that contribute more to the realignment loss are sampled more frequently in subsequent batches, and the optimal distribution can evolve throughout training," the authors write.
The mechanism is an "inner-outer optimization loop with a small overhead". The abstract does not quantify that overhead or break the gains down by language, and the headline figures are averages across the task suite. Code is on GitHub.
Shared on Bluesky by 2 AI experts
Originally reported by arxiv.org
Read the original article →Original headline: Online Language Adaptive Sampling for Better Distributed Cross-lingual Gains