Abstract
<p>This review employs a forensic methodological audit of 14 meta-analyses making broad claims about the impact of AI on education, evaluating construct coherence, primary study selection, and statistical validity. A brief review of meta-analytical methods establishes the best-practice benchmarks used in the audit. The audit found that none of the examined meta-analyses provided a valid basis for the claims they advanced: none had a coherent construct, none sufficiently assessed publication bias, and all had severe heterogeneity. Statistics were misapplied, miscalculated, and misinterpreted. A majority of the randomly vetted primary studies were problematic, most commonly because the outcome measured didn’t match the meta-analysis. Publication bias is well-documented in education literature, especially in edtech. Given a positively selected education literature, a meta-analysis with weak screening and inadequate publication-bias correction is aggregating publication bias and calling it evidence. These failures are consequential in a policy environment where AI adoption is increasingly treated as urgent and inevitable. The audited meta-analyses are heavily cited, with over 2,000 combined citations despite having been published for only 16 months on average as of this writing. These findings show how AIED meta-analysis can manufacture certainty from an evidence base that is heterogeneous, positively selected, and weakly vetted. The recurrence of severe validity failures across peer-reviewed journals points to a failure of editorial and reviewer gatekeeping, not isolated author error. Requirements for data sharing and stronger editorial and reviewer standards are recommended for meta-analyses in education journals.</p>