huggingface.co web signal

POSTECH: LLMs Writing Weights by Hand Skip Transformers

TL;DR

  • All three frontier LLMs tested (Claude Opus 5.5, GPT-6.1 Sol, and GPT-6 Astra) converge on sparse lookup tables when free to design their own architecture.
  • The best LLM-designed model hits 54.4% on the ARC benchmark with 3.35M parameters; a 3.33M-parameter transformer trained by gradient descent scored 29.8%.
  • The resulting associative-memory model runs up to 17× faster on GPUs than a same-sized transformer, and each answer traces to one editable fact row.

Given ten labeled questions, a sandbox, and the freedom to design whatever neural architecture they wanted, three frontier LLMs all picked the same thing. Not a transformer. A sparse lookup table.

A paper from POSTECH researchers Hyunjong Ok and Jaeho Lee, posted 18 November and surfaced via Hugging Face papers, asks a narrow question with a blunt framing. If a language model is writing neural network weights by hand, with no gradient descent in the loop, do the usual reasons for choosing a transformer still apply? Their answer is no.

When Claude Opus 5.5, GPT-6.1 Sol, and GPT-6 Astra were free to design the architecture themselves, the abstract reports, "all three compile plain-text facts into a sparse table queried by a fixed procedure, reaching up to 54.4% test accuracy" on the ARC science-question benchmark. The paper states the LLM-designed model "achieves higher test accuracy than the most accurate released Transformer that the same LLM wrote while using about 1,800× fewer parameters."

The numbers cut several ways. Claude Opus 5.5, by generation five, produced a 3.35M-parameter sparse-table model that hit 54.4% on ARC. A two-layer transformer at the same size, trained by gradient descent, scored 29.8%. Pretrained baselines remain far ahead: Qwen3.5-0.8B sits at 82.1%.

Two structural properties of the chosen design matter beyond raw accuracy. Each prediction traces back to a specific written row, and in the authors' words, "removing a row changes about one other answer." From this observation they propose an associative-memory model they report as "up to 17× faster on GPUs than a same-sized Transformer." Code is public on the authors' GitHub, landing in a stretch where our tracker has logged 386 open-source AI stories over the past 90 days.

The scope is narrow. MMLU accuracy sits near chance at roughly 25%. CIFAR-10 from raw pixels lands between 21 and 28%. As Ok and Lee put it: "pretrained LLMs remain more accurate; the claim concerns which structure to build when writing weights."