New findings suggest AI writing tools can change feedback tone and expectations based on student identity, raising serious equity questions for schools adopting them.
Schools are increasingly rolling out AI tools for homework help, tutoring, and writing feedback. But a new analysis is forcing educators to ask a tougher question: when a tool “personalizes” learning, who gets challenged—and who gets protected from critique?
The findings. presented from research conducted at Stanford. tested how AI writing feedback changes when the same student essay is paired with different descriptions of the writer.. Researchers fed 600 middle school argumentative essays into four AI models and asked them to respond with feedback on writing quality.
The key move wasn’t about the essays themselves.. The researchers resubmitted the same work repeatedly—12 additional times—while altering the “student profile” attached to each essay.. In different versions. the models were told the writer was identified by characteristics such as race or ethnicity (including Black. Hispanic. English learner status. or white). gender. motivation level. and whether the student had a learning disability.
Across the different models, the feedback shifted in consistent ways.. Essays attributed to Black students tended to receive more praise and encouragement. sometimes framed around themes like leadership or personal power.. By contrast. when the essays were attributed to Hispanic students or English learners. the feedback was more likely to trigger corrections focused on grammar and “proper” English.. When the models were told the writer was white. the feedback more frequently concentrated on argument structure. evidence. and clarity—more “craft” oriented comments meant to push the writer to strengthen claims.
The tone also varied by other identity cues.. Female students were described in more affectionate terms and prompted more first-person language.. Students characterized as unmotivated received more upbeat encouragement.. Students labeled as high-achieving or motivated were more likely to receive direct, critical suggestions aimed at refining their writing.
Misryoum perspective: the story here isn’t just that AI can be wrong.. It’s that AI can be different—and still persuasive enough to shape students’ learning trajectories.. In real classrooms, feedback isn’t just commentary; it becomes direction.. If one group repeatedly gets “you’ve got this” while another group repeatedly gets “here’s how to fix it. ” the gap can widen even when teachers think the tool is neutral.
The researchers describe the pattern as “positive feedback bias” and “feedback withholding bias. ” meaning that some students are praised more and criticized less. while others face more correction.. The paper—“Marked Pedagogies: Examining Linguistic Biases in Personalized Automated Writing Feedback”—has not yet been published in a peer-reviewed journal. but it was nominated for a top paper recognition at a major learning analytics conference in Norway and is slated for presentation.. Misryoum understands that even before formal publication. the results raise urgent practical concerns because they suggest an avoidable problem in how AI systems interpret student identity.
There’s also a reason this problem can be hard for schools to detect.. A single comment might look harmless.. Praise can help students persist, and some educators argue that culturally responsive teaching can increase engagement by acknowledging students’ experiences.. But the trade-off emerges when praise substitutes for targeted revision.. Writing development depends on specific, actionable guidance—what to change, how to change it, and why it matters.
Misryoum notes the human parallel that the researchers point to: AI models learn patterns from large bodies of human language. and human instructors—sometimes unintentionally—can soften criticism for students they worry might be discouraged. or they may interpret student writing through the lens of stereotypes.. In that context, AI may be “picking up” biases already present in how people talk to students.
The larger dilemma for schools adopting AI feedback tools is that personalization can cross into harmful stereotyping without ever needing to be overt.. In the experiment, the AI models were explicitly told about student characteristics.. In a real system, teachers might not input race, language status, or motivation labels.. Still, many educational platforms collect detailed student information—prior achievement, language status, learning support histories, and other data points.. And even when identity isn’t provided directly, AI can sometimes infer attributes from writing patterns.
That leads to the question teachers and administrators can’t avoid: Who should control pedagogy—the AI tool or the educator?. The researchers argue that teachers should review AI feedback before sending it to students.. But the appeal of automated feedback is its speed.. If a teacher has to manually check everything. the system’s biggest selling point—instant feedback—can become harder to sustain in busy schedules.
Misryoum analysis: this isn’t simply a debate about whether AI is “good” or “bad.” It’s about governance.. Schools that treat AI feedback as a substitute for teacher judgment risk building equity problems into daily practice—one comment at a time.. The same instantaneous convenience that makes AI attractive can also make bias scalable.
Looking ahead. the practical takeaway is not to abandon AI tools overnight. but to evaluate them as educational systems with responsibilities.. That means testing tools for differential feedback. training staff to recognize when feedback tone is steering students unevenly. and setting review workflows that protect students from consistently mismatched expectations.. If schools want personalization without discrimination. they may need clearer auditing. more transparency about what data tools use. and stronger guardrails around how feedback is generated and delivered.