Better learning with AI? Look closely at what the studies actually tested
Teachers need to know what helps a learner, not just what looks promising in a headline. A review of 477 learners favours technology-supported teaching, but only a few of its studies explicitly tested AI.
The short version
- What happened
- The paper pooled 11 studies and 477 participants, reporting Hedges' g=0.89 with a 95% confidence interval from 0.76 to 1.10.
- Who it affects
- Teachers can treat the result as a prompt to test specific feedback-rich activities, not as proof that an AI product will raise attainment.
- What remains unproven
- Only 11 studies and 477 participants were pooled, with high heterogeneity (I²=86.8%).
Keep in mind
Only 11 studies and 477 participants were pooled, with high heterogeneity (I²=86.8%).
Only a few studies explicitly tested AI; the estimate mainly concerns broader blended and digital instruction.
Outcomes were heterogeneous and should not be interpreted as a writing-specific effect.
The declared publication window and exclusion criteria do not align cleanly with every included-table entry.
The review included published English-language studies only, leaving publication and language bias possible.
Source analysis · 1 linked source. Primary analysed source date: 13 October 2026.
Read the full report · 6 min The evidence, methods, implications and limitations
A promising result, with a catch for teachers
A new review reports strong learning gains from teaching supported by technology. It offers a less certain answer to a question many teachers face: does adding AI help pupils learn? Only a few of the studies it combined explicitly tested an AI intervention.
The peer-reviewed meta-analysis, published in Frontiers in Education, combined 11 studies involving 477 learners. It reports a large pooled advantage for blended and technology-enhanced instruction over traditional teaching: Hedges' g=0.89, with a reported 95% confidence interval from 0.76 to 1.10. This standardised effect describes the combined studies; it does not isolate the benefit of an AI system.
The paper itself supplies the decisive qualification: only a few included studies explicitly incorporated AI-driven interventions. Others used learning-management systems, mobile applications, podcasts, vocabulary tools or broader blended-learning designs. The pooled outcomes also ranged from academic achievement and literacy to vocabulary, digital literacy and motivation. The result therefore describes a mixed family of technology-supported teaching approaches rather than a single AI intervention or a direct improvement in writing proficiency.[1]
What the researchers searched and combined
The authors searched ERIC, Scopus, Web of Science and Google Scholar and manually checked journals and reference lists. They report finding 510 records, removing 112 duplicates, screening 398 titles and abstracts and assessing 78 full texts. Eleven studies met their final criteria. Two coders screened the literature, with reported agreement of Cohen's kappa=0.87, and extracted means, standard deviations or test statistics used to calculate Hedges' g.
Seven of the 11 studies used quasi-experimental pretest–posttest control-group designs; the remainder included other experimental or single-group pre–post designs allowed by the protocol. Individual samples ranged from 23 to 94 learners, with a mean of about 43.4. The studies covered primary, secondary and tertiary settings in Indonesia, Thailand, Malaysia, China, Australia and the United States, with most coming from Southeast Asia.[1]
The pooled number hides major variation
The reported study effects ranged from 0.50 to 1.81, and heterogeneity was high: Cochran's Q was 68.42 on 10 degrees of freedom, p<0.001, and I² was 86.8%. A random-effects model is appropriate when true effects differ, but it does not make the interventions or outcomes interchangeable. High heterogeneity means the average should not be read as the result a teacher can expect from a particular product, class or group of students.
Exploratory subgroup estimates were larger in primary education than higher education, for four-to-eight-week interventions than shorter or longer programmes, and for interactive tools than static materials. Yet those comparisons split only 11 studies into still smaller groups. The authors correctly label them exploratory and warn that estimates may be unstable and underpowered. They cannot establish an optimum programme length or prove that interactivity itself caused the differences.[1]
Two eligibility tensions need resolution
The methods say the search covered studies published from 2020 through 2023, but the included-study table lists Xie et al. as a 2024 study. The eligibility section also says meta-analyses were excluded, while another included citation—Rohmawati, Holisin and Kristanti—has a title identifying it as a blended-learning meta-analysis. These may be reporting or citation errors rather than flaws in the underlying calculation, but the published article does not reconcile them.
That matters because a meta-analysis depends on a transparent, reproducible boundary around the studies being pooled. Including a study outside the declared period or combining a prior meta-analysis as though it were a primary experiment could change weighting and double-count evidence. A correction, supplementary extraction table and accessible analysis file would allow readers to determine whether the issue is editorial wording, a miscitation or a substantive eligibility problem.[1]
What teachers and school leaders can use now
The defensible lesson is that purposeful digital and blended instruction can outperform a comparison condition in small studies, especially when learners receive interaction and feedback. That supports careful local trials of tools tied to a defined learning objective. It does not justify buying an AI writing system on the strength of g=0.89, because the pooled estimate is neither AI-specific nor writing-specific.
Teachers should ask what students actually did, who provided feedback, how much class time changed and whether the assessment measured durable independent learning. A programme may succeed because of structured practice, extra attention or novelty rather than automation. Schools also need to inspect privacy, accessibility, unequal access and the quality of generated feedback—questions a learning-outcome average cannot answer.[1]
What would change the assessment
Confidence in an AI-specific benefit would require a preregistered review that separates generative AI, automated feedback, adaptive tutoring, ordinary learning platforms and non-AI digital tools. Each outcome should be analysed within a comparable construct, with duplicate samples removed and study risk of bias graded. Larger trials should compare the same curriculum and teacher support with and without the AI component, then follow independent performance rather than only work produced with assistance.
For now, the new paper is useful as evidence that technology-supported teaching deserves continued study and as a warning against headline compression. Its large pooled effect belongs to a heterogeneous set of small digital-learning studies. It does not demonstrate that AI itself improves writing, that any named product is effective or that schools should replace professional judgement with a broad average.[1]
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Continue the story
Read next
Continue exploring meta-analysis: compare the finding and its evidence limits.
Does generative AI improve self-regulated learning?
A meta-analysis of 17 comparative studies found a positive average effect, but results varied sharply: heterogeneity reached 94.85% and the prediction interval included harm as well as benefit. Almost all evidence came from higher education.
Read the next reportExplore the full source and evidence record
What this means for people
- Teachers can treat the result as a prompt to test specific feedback-rich activities, not as proof that an AI product will raise attainment.
- Students need assessment of independent learning, privacy and accessibility alongside scores produced during technology-supported work.
- School leaders should require an intervention-specific comparator and reproducible evidence before purchasing or scaling a system.
Global context
The evidence spans several countries but is concentrated in small Asian studies and mixes educational levels, subjects and technologies. The pooled estimate is not automatically transferable to another curriculum, language, funding environment or classroom infrastructure.
What the evidence does not yet show
- Only 11 studies and 477 participants were pooled, with high heterogeneity (I²=86.8%).
- Only a few studies explicitly tested AI; the estimate mainly concerns broader blended and digital instruction.
- Outcomes were heterogeneous and should not be interpreted as a writing-specific effect.
- The declared publication window and exclusion criteria do not align cleanly with every included-table entry.
- The review included published English-language studies only, leaving publication and language bias possible.
What to watch next
- A correction or supplementary table resolving the 2024 study and apparent meta-analysis entry
- Preregistered syntheses that separate AI from ordinary digital and blended instruction
- Larger controlled trials measuring independent, durable learning and teacher workload
- Transparent risk-of-bias ratings, extraction data and sensitivity analyses
Living evidence record
Impact record IAI-1FYBM3F
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
13 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what Frontiers in Education published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 13 October 2026
Research question, topics and regions
Whether a meta-analysis of blended, digital and partly AI-supported instruction establishes an AI-specific improvement in student learning
Topics: Education evidence · Blended learning · AI-supported teaching · Writing · Meta-analysis · Research quality
Regions: Asia · Global
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.