'Compile by Training' Reaches 83.6% on FuzzyBench-Hard Subset
TL;DR
- Compile by training reports 83.6% semantic accuracy on FuzzyBench-Hard, cases where the Program-as-Weights fast compiler produced no exact matches.
- Compile time runs about a minute versus seconds for the fast baseline, per the abstract's own trade-off.
- Teacher models generate task-specific training examples for a small adapter on a compact interpreter; the compiled function then runs without the teachers.
Compile by training reaches 83.6% semantic accuracy on FuzzyBench-Hard, a subset where the earlier Program-as-Weights fast compiler produced no exact matches. That is the headline number from a Hugging Face paper by Yuntian Deng, Pengyu Nie and Stuart Shieber.
The trade-off is compile time. It runs 'roughly a minute rather than seconds for the fast compiler,' the abstract says. At compile time, teacher models generate task-specific examples used to train a small adapter for a compact interpreter; the resulting function then runs without the teachers.
The authors frame the output as software rather than a service call: functions that can be 'stored, versioned, and composed like ordinary software.' The paper demonstrates three, described in the abstract as 'a multi-site website helper, a language-controlled 3D avatar, and a bidirectional English-Claudish translator.' It arrives during a busy stretch for open-source AI on our tracker, which has logged 272 stories in the category over the last 90 days.
Originally reported by huggingface.co
Read the original article →Original headline: Compile-by-Training Paper Turns Natural-Language Specs Into Small Local Neural Functions at 83.6% Accuracy