MultiHashFormer extends hash-token LMs to causal generation

A new arXiv preprint from Huiyin Xue, Atsuki Yamaguchi and Nikolaos Aletras takes an old trick from the encoder-only world and tries to make it work where it matters most for current LLMs, the causal, next-token side. The idea, described in the paper, is to stop representing each token as a row in a giant embedding matrix and instead encode it as a short sequence of discrete hash IDs produced by several independent hash functions.

The practical payoff the authors claim is a constant parameter footprint even when the vocabulary is expanded, including to multilingual settings. That matters because in a standard Transformer the embedding and output projection scale linearly with vocabulary size, which is a real reason multilingual frontier models are expensive and why labs end up making awkward compromises about which scripts to support. The architecture pairs a Hash Encoder that compresses the signature into a latent vector with a Hash Decoder that regenerates it for next-token prediction, which is how the team gets hashing to play with autoregression at all.

The headline empirical claim is that the model consistently outperforms standard Transformer language models at 100M, 1B and 3B parameters across multiple benchmarks. Take that as reported, not settled. The honest caveat is that this is a single team's preprint, currently under review, and what I pulled from the abstract does not give exact benchmark deltas, the specific language mix, or how the approach behaves on tokenisation-sensitive workloads like code or math, where hash collisions could plausibly hurt more than on average perplexity.

If the result holds up under independent replication, the people who benefit most are not the frontier labs, who already pay the vocabulary tax happily, but the open-source and academic groups building multilingual models on a budget, and anyone trying to ship one model that genuinely covers long-tail languages instead of one model per script. That is the direction worth watching here.