Audit: 14 AI-in-Education Meta-Analyses Fail Methodology Check

Fourteen meta-analyses that advance broad claims about AI's effect on education fail methodological inspection, according to a preprint posted to EdArXiv titled 'Presumed Effective: The Manufacturing of an Evidence Base for AI-in-Education Through Flawed Meta-Analysis,' discussed in a YouTube walkthrough of the paper.

The audit reports that "none had a coherent construct, none sufficiently assessed publication bias, and all had severe heterogeneity." Of the fourteen, 11 had no population boundary, 6 included a paper with fabricated references, and 3 included a study citing ChatGPT articles that predate ChatGPT itself. A majority of the randomly vetted primary studies were problematic, most commonly because the outcome measured didn't match the meta-analysis.

The papers under audit carry "over 2,000 combined citations despite having been published for only 16 months on average." The core charge: "a meta-analysis with weak screening and inadequate publication-bias correction is aggregating publication bias and calling it evidence." Two researchers in our Who's Who tracker surfaced the preprint link into our feed.