PaperCompiler Lifts Paper-to-Code Fidelity 13.8% on Paper2CodeBench

A new paper-to-code framework called PaperCompiler cuts high-severity evaluator critiques of the code it generates from 13.2% to 6.1%, according to an arXiv paper posted September 2 by Yunhao Liu, Hong Phuc Pham, and Jaehong Yoon.

On Paper2CodeBench, the same system posts a 13.8% relative improvement in reference-based fidelity, taking the score from 3.64 to 4.15.

The authors argue that today's paper-to-code agents leak information between stages. Intermediate outputs, they write, "are often presented as free-form plans or summaries that downstream coding agents may ignore, reinterpret, or compress, leading to algorithmic simplification and inconsistent repository structure." PaperCompiler's answer is a compilation step that turns paper evidence into "explicit repository-level implementation specifications" and labels each piece as paper-supported, inferred, externally delegated, or unresolved.

Those specifications, per the abstract, encode "non-degradation requirements, ownership assignments, cross-file dependencies, and file-level constraints." Repository generation then runs under the compiled spec while keeping local engineering choices flexible.

The abstract names no baselines, gives no per-venue breakdown across the ICLR, ICML and NeurIPS 2024 papers that make up Paper2CodeBench, and reports no compute or wall-clock cost for the compilation step. It arrives in the same week as BAAI's DisCo and a harness-of-harness result, the third piece of coding-agent research on our tracker in three days.