SchrodingerRepo scrambles repo cues, SWE-bench scores fall

A paper posted to arXiv argues that when you scramble the surface of a SWE-bench repository while keeping its behavior identical, coding agents get worse at it. The authors, from a group led by Silin Chen and Xiaodong Gu, call the framework SchrodingerRepo, and the pitch is in the name: the test repository only exists in a definite form once the agent walks into the evaluation environment.

The mechanism is four transformations stacked on top of the original benchmark task: "problem statement reconstruction, namespace remapping, intra-file layout reordering, and functionality-preserving code rewriting." Executable behavior is preserved. The naming, the file map, and the idiomatic shape a model may have seen during pretraining are not.

Run against SWE-bench Verified and SWE-QA, the effect is one-directional. "Removing familiar repository cues consistently degrades agent performance and substantially increases interaction costs across models," the abstract reports. The authors trace the extra cost to a specific stage of the agent loop: "the additional cost is primarily caused by increased difficulty in repository exploration and localization."

The conclusion the paper draws from that pattern is careful. Current coding agents "may partially rely on memorized repository-side cues," which is a softer claim than saying the leaderboards are wrong, but a pointed one given how much of the industry treats SWE-bench Verified as the reference score for agent progress. The abstract publishes no per-model numbers, no ranked table of which frontier systems degrade most, and no list of which of the four transformations does the heavy lifting. Those are the numbers that would tell a buyer how much of a given vendor's score to discount.