No spam, ever. We'll never share your email address and you can opt out at any time. Already a subscriber? Log in
arxiv.org/abs/2605.22705 arxiv.org/abs/2605.22821 Happy Linear Programming for Tokenization day! I was involved with two separate papers that hit ArXiv yesterday, using LP's to find the vocabulary maximizing compression, depending on the kind of inference you want to use.
arxiv.org/abs/2605.22705 arxiv.org/abs/2605.22821 Happy Linear Programming for Tokenization day! I was involved with two separate papers that hit ArXiv yesterday, using LP's to find the vocabulary maximizing compression, depending on the kind of inference you want to use.
Where does this happen? How does this happen? We examine this starting with injecting the representation of "or" from ["po", "or"] into the "or" from ["err", "or"] and see when the invasive meaning takes over; then we go even deeper... let us know what you think! arxiv.org/abs…
Try it on CharBench and let us know the results :) huggingface.co/datasets/omr...
It's been a long time since I reconnected with my interpretability past; here's our new analysis of the phenomenon of "detokenization": models reconstruct full word representations ("error") at the last state of the subword-tokenized location ["err", "or"].
Center = Yuval Pinter. Left = members they follow (green edges). Right = members who follow them (blue edges). Top = mutual follows (orange edges, slightly larger). Drag any node to reposition; click to open that profile.
