RepoFactory Lifts Qwen3.5-35B 22.6 Points on NL2Repo-Bench
TL;DR
- A Beihang, SJTU and IQuest Research team converted real GitHub repositories into 10,159 executable training environments spanning 17 software domains.
- Fine-tuning Qwen3.5-35B-A3B on the set lifts NL2Repo-Bench from 23.8 to 46.4, the top listed open-weight result below 200B parameters.
- An Environment Evolution Loop driven by static code metrics, not solver rollouts, contributes 5.1 of the NL2Repo points and 2.1 points on a five-benchmark average.
Fine-tuning Qwen3.5-35B-A3B on 10,159 synthesized coding environments raises its NL2Repo-Bench score from 23.8 to 46.4, a 22.6-point gain. The result comes from a paper by Beihang University, Shanghai Jiao Tong University, and IQuest Research posted to Hugging Face. It puts their 35B result above GPT-5.5 at 45.1 and Gemini-3.1 Pro at 33.4 on the same benchmark, and the authors call it the strongest NL2Repo score they list for any open-weight model below 200B parameters.
The pipeline, called RepoFactory, turns real GitHub projects into containers where an agent is handed a language spec and asked to rebuild the repository from scratch. The paper describes the setting as one in which "an agent must build a complete, functioning repository from scratch given a language specification." Candidates must have package metadata and a non-empty test suite, at least five implementation files, and fewer than 40,000 lines of Python; projects requiring outbound network access, graphical interfaces, or specialized hardware are excluded. Across the 10,159 validated instances the verifier contains 2.09 million statically identified test definitions.
The twist is an Environment Evolution Loop that iterates each candidate for up to 12 rounds, revising oracle code, spec, and tests to maximize a "multiplicative objective combining validity, structural coupling, and implementation scope through proxy feedback." Those are static code metrics, not solver rollouts, so the loop avoids running a model on every candidate. Ablating the loop costs the final checkpoint 5.1 points on NL2Repo and 2.1 points on the five-benchmark average.
The transfer story is thinner. SWE-bench Verified moves from 65.8 to 72.2, SWE-bench Multilingual gains 1.3 points, and Terminal-Bench 2.1 lifts 6.8. The paper evaluates with Claude Code version 2.1.140 as the agent scaffold, a 262,144-token context, and temperature 1.0. No per-domain breakdown is given for the 17 software domains the training set claims to cover, and the Python-only filter sits awkwardly next to the modest Multilingual result.
The dataset and the RepoHorizen-35B-A3B checkpoint are on Hugging Face; code is at github.com/IQuestLab/RepoFactory. The paper arrives during a stretch when our coding-tools tracker has been logging new agent training infrastructure almost every day.
Originally reported by huggingface.co
Read the original article →Original headline: RepoFactory Turns 10,159 Real Repos Into Coding-Agent Training Environments, Lifts NL2Repo-Bench 22.6 Points