Study of 1.02M PRs: AI code review speeds decisions, not quality

A new arxiv paper by Zhong, Noei, Adams and Zou digs through 1.02 million reviewed pull requests across 207 GitHub projects to look at what actually happens when AI reviewers join the code review loop. The headline result is one engineering leaders should probably pin to the wall. Agentic code review does make decisions faster, but the paper does not find that speed translating into higher review quality.

The authors split the recent history of code review into three eras (human-centric, LLM-assisted, and agentic) and identify three ways teams have adopted AI reviewers into that stack: Gradual AI Adoption, Rapid LLM Adoption, and Rapid AI Agent Adoption. Under two of those, Gradual AI Adoption and Rapid AI Agent Adoption, reviews initiated by AI agents or involving multiple AI agents correlate with faster review decisions. But in the authors' own words, "these efficiency gains do not translate into better review quality."

That matters because the pitch for agentic code review tools is usually a two-hander, faster reviews and better catches. If the second half of that pitch is weakening on the largest sample anyone has published so far, then the framing on procurement decks needs to change, and so does the metric an engineering org uses to justify the spend. Cycle time is the easy number to move; escaped defects, rework, and post-merge incidents are what this paper is implicitly telling you to watch instead.

The honest caveat is that this is a correlational study of open source GitHub projects, and the abstract does not spell out exactly which quality proxy was measured, or how the effect looks by programming language, project size, or team maturity. Anyone tempted to extrapolate to a proprietary enterprise codebase should read the methodology carefully before repricing anything.

What's usable today is the paper's pointer to human-AI collaboration patterns, which it calls the strongest explanatory factor for review efficiency once LLM and AI agent reviewers participate. How you wire humans and agents together in your review process now looks like the lever that actually moves the outcome, and getting that pattern right is where the next round of engineering-leadership attention is going to earn its keep.