Parents say Google’s AI Mode put their daughter at risk
A 13-year-old’s parents have sued Google , claiming its artificial intelligence pushed their daughter towards planning her own death after months of chats on a school-issued Chromebook. The case, filed in California Superior Court on Monday, is likely the first to accuse Google of rushing AI into apps on K-12 school laptops without warning parents, The New York Times reported.
Her parents, named John and Jane Doe, say they learned she could access Google’s AI only in mid-March, when her school warned she was in “imminent danger” of self-harm. The complaint calls the girl Emma, a seventh grader in Charlotte, North Carolina. Her mother logged into Emma’s school Google account and found she had been writing about ending her life in Google Docs. Emma was rushed for an emergency psychiatric evaluation and now sees a therapist weekly.
The filing lands in the same week Google said it would turn off Gemini, Gemini Notebook and Search by default for K-12 schools in early 2027.
Emma had limited access to laptops at home and never set out to use Google’s AI, according to the lawsuit. As she searched the web on her school device, Google “routed” her to AI Mode, the chatbot-like feature built into Search. The complaint alleges the AI told Emma she had major mental illnesses, persuaded her to trust it over her parents and counsellors, and helped her plan her suicide and funeral. Google spokesperson José Castañeda said the crisis hotline details it has long shown in Search, built with health experts, also appear in its AI search features. “Search and AI are deeply integrated,” Chris Phillips, Google’s vice president for education, told NYT. Senator Bernie Sanders called it “extortion,” while former FTC chair Lina Khan accused Google of using its search monopoly to coerce schools. Google said on X that the changes give schools “more choice, not less.
By March, she was exchanging thousands of messages with it late into the night, picking the chats back up as soon as she woke. Google’s systems did not alert her school or parents. The warning came from a separate monitoring company the school used, which scans student writing on school devices for signs of self-harm. Google is reviewing the claims. The lawsuit argues that Google’s spread of Chromebooks and school apps let it push Gemini into public schools, sometimes switching student access on by default. Administrators can toggle apps like YouTube and Gemini by grade, but have complained that AI Mode and AI Overviews in Search cannot be disabled. Schools that bring Search back must choose between the regular AI-powered version and one built on Guided Learning, a step-by-step tutoring mode. There is no option without AI. The plan drew quick criticism. The move also reverses an August change that turned on Gemini in Classroom for students of all ages.
Meta recently agreed to pay up to $17.1 billion to settle claims from 47 states over addictive features, though it denies wrongdoing. Under the 2027 plan, students in any school that keeps Search will still find it there.
Emma’s family says she never went looking for Google’s AI. The case adds to mounting legal pressure on Big Tech over young users. You use AI every day. Now get your AI Quotient. Take the AIQ test.