Our new paper reformulates tokenisation as a linear program (LP), which we solve to get SOTA tokenisers 😁 As a bonus, this LP tells us how close to optimal any tokeniser is! Check it out 👇 w/ J. Tempus, @philipwitti.bsky.social, @craigschmidt.com, D. Komm Paper: arxiv.org/abs/…
Who's Who of AI
Tiago Pimentel
Why they matter
Researcher with public evidence across AI research.
- AI signals
- 2 past 30d
- Sources
- 2 distinct domains
- Discussões
- 0 past 30d
- Latest signal
- 18d ago
Postdoc at ETH. Formerly, PhD student at the University of Cambridge :)
What they're sharing
[2605.22821] Tokenisation via Convex Relaxations arxiv.org
What Language is This? Ask Your Tokenizer arxiv.org
Validating Causal Abstraction Metrics on Simulated Complex Systems melouxm.github.io
View on Bluesky →
Articles & links
Tiago Pimentel reposted
Effective language identification based on a tokenizer UnigramLM tokenizer already gives probabilities, testing those to identify a language is fast and effective. Whiceh leads me to wonder, can we identify language during training and affect behavior? arxiv.org/abs/2602.17655…
AI Weekly's analysis
→
- UniLID reuses the UnigramLM tokenization algorithm to predict a string's language by asking under which language's unigram distribution the string is most likely.
- The method reaches roughly 70% accuracy with as few as five labeled samples per language in low-resource settings.
- It supports incremental addition of new languages without retraining and integrates into existing language model tokenization pipelines.
Read full analysis →
View on Bluesky →
Tiago Pimentel reposted
I'm very happy to give a spotlight at the Mechanistic Interpretability Workshop @ ICML on our work: "Validating Causal Abstraction Metrics on Simulated Complex Systems" Which metrics actually tell you if an explanation is valid? We built a benchmark to find out. 1/n
Their network
In Tiago Pimentel's orbit
Center = Tiago Pimentel. Left = members they follow (green edges). Right = members who follow them (blue edges). Top = mutual follows (orange edges, slightly larger). Drag any node to reposition; click to open that profile.