Curriculum and edtech providers are rapidly incorporating artificial intelligence (AI) into instructional materials with little or no evidence that the AI add-ons improve student learning, according to a new brief from EdReports, a nonprofit that reviews pre-K-12 instructional materials. Only one of the 10 vendors in the study included third-party evidence that the AI features they embedded improved educational outcomes.
“I knew that there wasn’t going to be much evidence, but I was surprised at how little there was,” says Courtney Allison, chief academic officer for EdReports. She says the report underscores a growing challenge: behind-the-scenes AI updates make it difficult for schools to ensure product efficacy.
“This has been a problem for a while with digital tools, and it’s only accelerated with AI,” echoes Brittany Miller, executive director of the Center for Outcomes Based Contracting, an organization that helps school districts link edtech procurement directly to academic outcomes. “Tools are growing quickly, and the market is getting crowded and confusing.”
Sara Kloek, vice president of education and youth policy with the Software and Information Industry Association (SIIA), notes that the research “just may not be there yet” for some of the vendors incorporating AI into educator- or student-facing products.
The Study and Impact
For the qualitative study, EdReports selected 10 K-12 curriculum and edtech providers. Four were “among the top 10 publishers” of purchased instructional materials in the country, three were mid-sized curriculum providers and three were digital-first edtech companies. EdReports reviewed each company’s AI-enabled products, key features and claims about their use. The researchers notified the companies about the review and kept their identities anonymous.
The blurred lines between AI-enabled course materials and materials without AI can affect classrooms, observes Jean-Claude Brizard, president and CEO of Digital Promise, a global nonprofit working to expand equitable education opportunities. The worst-case scenario, he says, is that “quality assurance at adoption no longer guarantees quality at instruction.”
Even identifying whether instructional materials contain AI can be challenging, since it can be introduced without notification from the vendor. “Something as mundane as understanding where that communication was from the vendor that had the new product updates” becomes crucial, says Lora Kaiser, executive director of the Center for Education Market Dynamics, a nonprofit market intelligence organization that aims to improve outcomes for underserved students by expanding the adoption of high-quality instructional materials.
Kaiser wants vendors to give schools a choice about accepting any new AI updates. “Is it a feature I can opt in to?” she says. “For our teachers and our students, anytime that little AI enhancement pops up, that’s an instructional decision. We should know, as a fact, that it’s grounded in evidence, and understand how it impacts instructional integrity for students.”
The “Evidence” Gap
The EdReports brief also states that while existing frameworks frequently analyzed AI integrations for privacy and safety, there was a lack of consistent ways to determine if AI can strengthen learning or preserve curriculum coherence. This can present a roadblock to creating evidence of educational efficacy.
A lack of agreement on what “evidence” means in the first place compounds the problem, says Katie Boody Adorno, founder and CEO of Leanlab Education, a nonprofit that codesigns research between edtech companies and schools.
“I think the biggest takeaway from the report is that we need to move away from just checking a box around safety and evaluation, and more toward quality: What are the arbiters of quality, and how are we measuring and communicating that?” she asks.
Matt Moyer, principal of Rupert Elementary School in Pottstown, Pennsylvania, says his school uses some edtech tools with AI embedded in them. “My take on tools like that is there’s some good in them, but I’m not yet convinced they lead to results outside the usage of that tool itself, which I think is what the report was getting at,” he says.
“What’s been interesting is realizing, over the last couple of years, that some tools we were already using had AI built in that we simply hadn’t been told about,” Moyer continues. Even when product pitches from edtech vendors do mention AI, he says the conversations are usually built around “what’s best about their AI compared to someone else’s.”
While educators want evidence that course materials actually work, Miller of the Center for Outcomes Based Contracting points out a crucial missing piece: how those tools are implemented.
“The study doesn’t really account for what a student actually does when they engage with the materials,” she says. “I think it’s a matter of sending the right signals from the buyer side to the market: You actually have to help us implement this more effectively, or we’ll see the exact same pattern we’ve seen over the last 20 years — a shiny new object that doesn’t get implemented well, which is why we can’t build evidence.”
How Vendors and Schools Can Respond
Kloek advises edtech vendors to update their products to align with the latest cybersecurity best practices. She recommends SIIA’s seven Principles for AI in Education, a framework to guide the edtech industry’s implementation of artificial intelligence.
The principles suggest that AI technology prioritizes instructional needs, educational equality, student data protection and transparency. The principles also suggest that edtech companies that are building AI tools should engage with education institutions on new AI developments, adopt best ethical practices for accountability and work to improve AI literacy among students and educators.
Kloek says SIIA is looking to update the principles to include additional transparency documents for edtech on how to communicate to school districts and parents what their products do and the purpose behind the tools.
“Ultimately, though, the district decides how these tools are used and can decide whether that’s appropriate and aligns with the district’s plans for AI or for technology generally,” Kloek says.
Leanlab Education’s Katie Boody Adorno believes that edtech vendors need to offer more specific data to back up their claims.
“You see a lot of companies saying [their products are] ‘based on cutting-edge research,’ meaning they built the tool around a loose interpretation of some foundational research, often from a university setting,” she says.
EdReports’ chief academic officer Courtney Allison suggests districts and schools start demanding more from their vendors.
“Buyers need to ask the question, ‘What does this badge or this membership or this certification actually demonstrate?’” she says. “They also must start the conversation by asking themselves, ‘What’s the instructional vision for the students?’”
Principal Matt Moyer believes evidence from an edtech vendor tends to raise more questions than it answers. He counsels school and district leaders to ask vendors about the sample size of the research they conducted about their product, the researchers and parameters of the study, and other pointed questions.
“Educational leaders should look at these claims and ask what they actually translate to, because I’ve seen plenty of cases where a number meant something different once you dug into it,” he says.
Digital Promise’s Jean-Claude Brizard suggests schools search online directories, such as the EdTech Index or LearnPlatform, for vetted providers and evaluate them for data privacy, usability and educational value.
“Current quality frameworks were built for fixed materials, like curricula,” Brizard explains. “As edtech and AI integration accelerates, we need a more dynamic system of measuring quality and efficacy.”