AI therapy tools flag alerts in Florida schools

AI therapy – In Florida’s Putnam County, a middle school counselor says an AI-enabled therapy platform flagged a “severe” risk chat last spring—triggering calls to a parent and police—while the district’s broader rollout is now raising a new, uneasy question for schools: c

At 7 p.m., Brittani Phillips checked her phone and saw an alert she couldn’t ignore.

Phillips is a middle school counselor in Putnam County, Florida. She receives messages from an artificial intelligence-enabled therapy platform that students use during nonschool hours. The system flags when a student may be at risk for harming themself or others based on what the student types into a chat. When the alert came, she says, it was “severe” for an eighth grader.

Phillips spent her evening on the phone with the student’s mom, probing what was going on and how vulnerable the student was. She also called the police. “He’s alive and well. He’s in ninth grade this year,” Phillips later said.

She credits the interaction with building trust. “When the student passes her in the hall now, he makes a point to greet her,” Phillips added.

That kind of outcome is one reason schools are increasingly looking at AI tools for student mental health support—especially when budgets tighten and counselors are stretched thin. But it is also at the center of a debate over something far more delicate than detection: what happens to students when help arrives through a chat window rather than a clinician’s office.

Interlachen Jr.-Sr. High School, where Phillips works, uses an AI platform to vet students’ mental health needs. Her district has used Alongside, an automated student monitoring system, for three years. The company markets its approach to K-12 districts. placing it in a rapidly expanding category of tools aimed at similar purposes—at least 9 companies have received funding deals since 2022.

Alongside says its tool is used by more than 200 schools across the United States. The company argues that its platform offers better services than typical telehealth options because it includes a social and emotional skill-building chat tool. Students “yak about their life-problems” with a llama called Kiwi, which is designed to teach resilience. Alongside also says its AI-generated content is monitored by clinicians. For schools with limited access to care—especially in rural areas—company representatives say the system offers resource-tapped districts a way to reach critical mental health support.

For Phillips, the value is immediate and practical. Navigating budget shortfalls and limited mental health staff. she says the tool is exceptional at putting out the “small fires.” With around 360 middle schoolers to support. she said having the platform allows her to hand-hold students through breakups and other routine problems that can turn dangerous as they accumulate. She also argues that some students find it easier to turn to AI for emotional problems.

And in her view, that matters because the alternative—reaching a counselor or a therapist—can feel out of reach when a teen is spiraling.

Digital rooms, not therapy rooms

Counselors point to a mix of anxiety and familiarity as part of why students confide in these technologies.

Sarah Caliboso-Soto. a licensed clinical social worker. assistant director of clinical programs at the USC Suzanne Dworak-Peck School of Social Work. and clinical director for the Trauma Recovery Center and Telebehavioral Health at USC. said speaking with a mental health professional can be intimidating for adolescents. She also tied comfort to generational habits: students who have grown up encountering chat interfaces through social media and websites may experience AI systems as familiar.

Linda Charmaraman. director of the Youth. Media & Wellbeing Research Lab at Wellesley Centers for Women. added that it can be easier to text than call someone on the phone. She also said using AI to work through emotions allows students to avoid watching facial expressions—something they may worry could signal judgment.

Chatbots, Charmaraman said, are also available at times when a human might not be available, without the hassle of appointments. “It’s almost more natural than interacting with another human being,” Caliboso-Soto said.

Caliboso-Soto has seen a rise in crisis text lines and chat lines through her work with a telehealth clinic. That clinic does not use AI. she said. but she has seen companies approach it with requests to use AI inside therapy sessions as notetakers. She said that isn’t necessarily bad in her view; for resource-strapped schools. she described AI as potentially “as a first line of defense. ” regularly checking in with students and pointing them toward more support when they need it.

Still, both she and Charmaraman worry about where the line should be drawn.

Caliboso-Soto said she worries about using AI as a substitute for a counselor. She said it lacks the discernment clinicians provide. and that even if large language models can be trained to notice symptoms in text. they cannot see or hear what a human clinician can—voice inflections. and movements of the body. She said they also cannot reliably catch subtle observations or behaviors. “You can’t replace human connection, human judgment,” she added.

Charmaraman echoed that concern, saying it’s crucial not to overly rely on AI for mental health. She said the technology can miss nuances a human counselor would catch, and can offer unrealistic positive reinforcement. She argued schools need a holistic approach that includes families and caregivers.

She also raised a practical question tied to systems that filter “serious” cases: if AI is used to intervene on more urgent issues, are students losing contact with clinically trained humans more often?

Alongside representatives dispute the idea that AI is meant to replace care. Ava Shropshire. a junior at Washington University who serves as a youth adviser for Alongside. said the app is not intended as a replacement for human therapy. The app, she said, is a stepping stone toward seeking help from adults. Shropshire said it makes mental health and social-emotional learning feel more normal for students and can lead them to seek human help.

Even so, not every student sees the tool as progress. The article notes that some students think it’s at best a Band-Aid.

A policy push alongside unease

The debate about AI in schools is happening inside broader political momentum, and inside families’ private living rooms.

AI is a major component of the Trump administration’s national education agenda, the article says. At the same time, some parents, educators, and lawmakers are wary of increasing teens’ time in front of screens. States have also started restricting the use of AI in telehealth.

Some families and advocates also worry about emotional attachment to chatbots. A recent national survey found that 20 percent of high schoolers have used AI romantically or know someone who has. There is also interest in keeping students from emotionally connecting with bots. including a proposed federal law that would force AI companies to remind students that chatbots aren’t real people.

These concerns aren’t abstract to those working with youth either. Sam Hiner. executive director of The Young People’s Alliance. a North Carolina-based organization that lobbies for more youth participation in politics and policymaking. described loneliness and weakened community connections as part of the backdrop for student tech use.

During a time of economic upheaval, Hiner said, technology and social media have manipulated and isolated students from one another, creating a deep yearning for community and belonging. He said students will get it wherever they can, even through ChatGPT.

The Young People’s Alliance released a framework for regulating AI that allows for some therapeutic uses of the technology. But the organization. Hiner said. is striving to rebuild human community and is against AI use when it threatens to replace human companionship. “That’s a critical aspect of therapy and of living a fulfilled life and having social connection and having mental well-being. ” he added.

For Hiner, the central risk has a name: a “parasocial relationship,” when students form a one-sided emotional attachment. He said it matters most when technology enters schools for therapeutic purposes. He said it may be valuable for AI to provide feedback or conduct analysis—even in mental health contexts—but argued that it should not imply it has its own emotional state. for example by saying “I’m proud of you” to a student user. Hiner said that kind of messaging encourages attachment.

He also criticized how platforms measure success. While platforms may claim to decrease loneliness. he said they do not really measure whether people become more connected over the long term. “All [tech platforms are] measuring is whether this bot is serving as an effective crutch for the immediate feelings of loneliness that they’re experiencing. ” he said.

What advocates want to prevent, Hiner said, is bots fueling the loss of social skills by pulling students away from relationships with other people—relationships that create social accountability.

Privacy and the messy boundaries of monitoring

Alongside’s use of AI chat tools also brings privacy questions. The article notes that these chatbots do not generally carry the same privacy protections as conversations with a licensed therapist. It adds that when concerns about student privacy and encounters with police are high. use of these tools raises “messy” privacy concerns even when supervised by clinicians. according to a privacy law expert.

Both Phillips and the company stressed that these systems need human oversight. Phillips described the platform as an improvement over other monitoring tools her district has used—tools that, in her view, tended to point students toward in-school discipline rather than mental health support.

This school year. Phillips said she noted 19 “severe” alerts from the AI health tool as of February. out of a total of 393 active users. Alongside does not separate the incidents by which students caused them. she said—meaning some of the same students are causing multiple of those 19 “severe alerts. ” Phillips added.

Phillips said she has also learned that it can take a human to understand teenage humor. She described moments when middle school students—usually boys—test the boundaries of the technology by typing lines such as “my uncle touches me” or “my mom beat me with a pole” into the chat to see whether she will follow up. She said these students were often trying to determine whether anyone is listening. trying to test whether anyone cares. or simply seeing whether it’s funny.

When she pulls them aside to discuss it. Phillips said she watches for body language changes that might suggest whether the comment was real. If it was a joke, she said, students often become apologetic. When a student doesn’t seem remorseful, Phillips said she calls and lets parents know what happened.

Even in cases where she believes the alert wasn’t a genuine report of harm, Phillips said she feels she has more options than other monitoring systems that would refer the student to in-school suspension.

Because Phillips pays close attention to the interactions, she said students also learn to trust that she is actually monitoring the system. She added that the number of boys who test the system in that way goes down every year.

AI in schools mental health school counseling chatbots student monitoring Alongside privacy telehealth K-12 education teens