AI policing – Two recent U.S. cases show how AI misidentification and overreliance on probabilities can escalate into gunpoint arrests and wrongful jail time.
A surveillance camera fueled by artificial intelligence misread a teenager’s pocket as a weapon. triggering a fast escalation that ended in handcuffs and a search that found only chips.. Months later. a facial recognition system linked a Tennessee grandmother to fraud charges in North Dakota—a state she had never visited—leading to five months behind bars.. Together. the cases underscore a central risk of AI in public safety: probabilistic outputs can be treated as certainty. with life-altering consequences.
In Baltimore on Oct.. 20. 2025. 17-year-old student Taki Allen was sitting outside his high school after football practice when an AI-enhanced surveillance camera falsely identified a Doritos bag in his pocket as a gun.. Police arrived within moments.. Officers drew their weapons. forced Allen to his knees. and handcuffed him while they searched him. only to find a crumpled bag of chips.. What began as an ordinary evening became a traumatic confrontation—driven by a misidentification and by human decisions that followed the system’s signal.
In another case, Angela Lipps, a Tennessee grandmother, was released on Dec.. 24, 2025 after spending five months in jail.. The arrest stemmed from facial recognition software that incorrectly connected her to fraud crimes in North Dakota. a location she had never visited.. Police arrested her at gunpoint while she was babysitting her four grandchildren.. The circumstances illustrate how technical errors can collide with police procedures that treat an algorithmic match as a starting point for action rather than a prompt for verification.
The underlying issue in both incidents is the same.. AI systems often produce probabilities—not definitive facts—and yet people can end up treating those outputs as certainties.. Researchers studying the intersection of technology. law and public administration describe a pattern that can be especially dangerous in policing: the rapid shift from a statistical signal to operational decision-making. where uncertainty disappears on the way from a model’s output to what officers ultimately do.
Police use AI tools in multiple cities, but there is no public registry that tracks the full extent of deployment.. In general terms. some systems ingest historical crime data and score neighborhoods on predicted risk to route officers toward places flagged as “hot spots.” The mechanism can seem straightforward: historical patterns are translated into scores and maps.. The problem is that once an AI system suggests a possible threat. the question often stops being whether the model’s output is reliable and becomes what action should follow immediately.. A statistical output turns into an operational trigger.
That distinction matters beyond these two examples.. When generative AI models respond to questions, they do not behave like a database that searches and verifies facts.. They generate likely answers based on patterns in training data.. Even when such responses are correct, they may not provide the full context behind the truth.. The danger, researchers warn, comes when users assume the system is retrieving verified information rather than producing a likelihood.
In policing. similar concerns apply to predictive tools that estimate where crime may occur based on geographical data and prior incidents.. Algorithms can generate risk scores or heat maps for locations that sound authoritative. but they may have limited relevance to who is involved in a new crime in that area.. Even when a system’s outputs are accurate about past trends. that does not necessarily convert into reliable evidence about present individuals.
Some researchers have argued that predictive policing systems may not increase racial disparities in arrests compared with traditional policing.. But the broader concern described by the researchers is not confined to whether arrest rates measurably shift.. Instead. it centers on how probabilistic predictions can become standardized operational decisions—potentially without further verification—effectively treating uncertainty as something officers must act on.
The researchers also note that caution about using AI “in isolation” has been raised by AI researchers and by work conducted in partnership with law enforcement leaders.. Studies involving the University of Virginia’s Digital Technology for Democracy Lab and police chiefs suggest that some agencies adopt strict policies defining how technology should be used alongside human discretion. while others have no such policy.. That variation can shape how often algorithms influence decisions and how much human judgment is allowed to challenge a system’s signal.
A key point is that AI systems rarely offer binary answers like yes or no.. Many models provide probabilities or confidence scores, and designers often set a threshold that determines when an alert is triggered.. Researchers describe this like a control knob: a threshold such as 95% reflects how likely the system’s interpretation is considered to be.. A low threshold can catch more potential threats while increasing false alarms.. A high threshold can reduce mistakes but risks missing genuine dangers.
What often remains invisible to the public, the researchers say, is that these thresholds are choices. Even when the code is written by vendors or adjusted by agencies, the selection of where to draw the line determines when algorithmic suspicion becomes real-world police action.
The trade-offs are clearer in other regulated domains like medicine. where diagnostic tools are calibrated to balance different errors. and where follow-up by professionals and oversight mechanisms can be part of the workflow.. In public safety, the same type of balancing is present, but it is frequently less transparent.
In law enforcement. the stakes include both false positives—when systems flag threats that do not exist—and false negatives—when systems fail to detect real danger.. The consequences of those errors are stark.. A lower threshold may produce more alerts and prompt earlier intervention. but it can also increase the risk of mistaken identifications like the one that led to Lipps’s arrest and detention. or escalated encounters like Allen’s.. A higher threshold may avoid some wrongful interventions but can leave legitimate risks unaddressed.
The researchers also describe how developers attempt to optimize thresholds using statistical techniques such as receiver operating characteristic curve analysis and precision–recall analysis.. These methods can evaluate how changing a threshold alters the balance between correctly identifying events and incorrectly flagging harmless ones.. Fine-tuning may improve performance. but it cannot fully resolve the societal question of how much uncertainty a community is willing to tolerate when police authority is on the line.
That question links directly to legal standards of proof.. Courts require different levels of evidence depending on the stakes—such as probable cause. preponderance of the evidence. or beyond a reasonable doubt—and these standards reflect judgments about how much uncertainty is acceptable before legal power is exercised.. A court is not meant to accept a guess or prediction; it must follow a structured process to weigh evidence.. In contrast, AI systems often do not express uncertainty in the way humans do.. A model may show confidence in a reply even when it is wrong. leaving users to decide whether to treat the output as something that requires verification.
As AI spreads further into public life—including the courtroom. schools. healthcare settings. and government decision-making—the researchers argue that the public needs to understand what AI can and cannot do.. The central message is that AI does not “know” things the way many assume.. It does not naturally distinguish between “maybe” and “definitely”; that interpretation has to be supplied by people.. They contend that technologists should design systems that acknowledge uncertainty and that users should be educated to interpret outputs responsibly—especially when the consequences can include escalation. detention. and wrongful convictions.
AI policing wrongful arrest facial recognition predictive policing legal standards of proof surveillance technology U.S. law enforcement policy
This is exactly why AI policing is terrifying. A system looks at pixels, guesses “gun,” and suddenly a teenager is on the ground with officers aiming at him. Nobody should be getting handcuffed because an algorithm treated a probability like a fact.
Mason Rodriguez, I agree with the core point, but the bigger issue is process. If they’re using AI outputs to trigger lethal-force posture, there needs to be hard rules: the AI can’t be the “trigger,” only a lead. Also, where was the independent corroboration before weapons were drawn?
Taki Allen and that Tennessee grandmother both sound like the same movie: “AI said so” turns into “oops, wrong person” after people lose freedom. And then we’ll get a statement about “continuous improvements.” Sure. Improving what—until the next wrong arrest?
I keep thinking about the grandmother part. Five months for fraud tied to a facial match in a state she never even visited… that’s not a minor glitch, that’s people paying for an algorithm’s mistake. If it can’t be reliably right, it shouldn’t be relied on at all.