Found first: a primary source the press has not covered yet.
Researchers from Shanghai Jiao Tong University, Xi'an Jiaotong University, and East China Normal University have introduced SchrodingerRepo, a framework that transforms test repositories at evaluation time to remove memorized structural cues. Their paper, tested against SWE-bench Verified, finds all four models assessed dropped in pass rate after transformation, with declines ranging from 6.0 to 14.4 percentage points.
What the source says
The authors, Silin Chen, Yufei Yang, Xiaodong Gu, Yuling Shi, Chengcheng Wan, and Haibing Guan, apply four transformation levels to repositories: problem-statement reconstruction, namespace remapping, intra-file layout reordering, and functionality-preserving code rewriting. On SWE-bench Verified, Gemini-3.1-Flash-Lite fell from 56.7% to 42.3% (14.4 pp drop); GPT-5.4-mini from 46.8% to 35.6% (11.2 pp); GPT 5.1 from 44.6% to 36.2% (8.4 pp); DeepSeek-v4-Flash from 72.8% to 66.8% (6.0 pp). The drops for GPT-5.4-mini, GPT 5.1, and DeepSeek-v4-Flash are reported as statistically significant at p<0.01. The authors conclude that models "may partially rely on memorized repository-side cues" including naming conventions, file layouts, and implementation patterns.
Why it matters
SWE-bench Verified is the benchmark the industry has converged on for comparing coding agents, and leaderboard positions directly inform deployment decisions. If pass rates are partly a function of familiarity with known repository structure rather than generalizable reasoning, those scores overstate likely performance on unfamiliar codebases. The problem is not confined to any one model: all four labs tested show measurable degradation, and the worst drop comes from a model that appeared near the top of its cohort on the unmodified benchmark. SchrodingerRepo is a practical counter, generating fresh transformations at evaluation time rather than relying on a fixed test set that models may have absorbed during training.