What Language is This? Ask Your Tokenizer
2 directory members surfaced this signal.
“Effective language identification based on a tokenizer UnigramLM tokenizer already gives probabilities, testing those to identify a language is fast and effective. Whiceh leads me to wonder, can we identify language during training and affect behavior? arxi…”