Anthropic staffers sound the alarm—again—on AI catatrophe

On Tuesday, experienced AI researcher Jacob Coxon resigned from the AI firm Anthropic—saying that both that company and OpenAI, his previous employer, were “gambling with our lives” by developing models that could improve themselves at a rapid clip until they reach “superintelligence.”

In an alarming social media post, Evan Hubinger, who leads a team that tries to stress-test Anthropic’s models for safety, essentially agreed that the company’s staffers “really do earnestly believe AI could kill all humans!” 

It’s far from the first time researchers have tried to raise the alarm about AI’s dangers, but Coxon’s post and subsequent discourse went viral. 

The details of AI alignment can be difficult to grasp—a big part of the problem is that no one truly understands the intricacies of how advanced models work. But it doesn’t take expert-level knowledge or insider secrets to understand the reasons for alarm. All you need is three facts about AI: It is already surpassing human abilities in important domains. Leading companies continue to improve it rapidly. And no one knows how to reliably keep its behavior in line with human goals.

Yesterday, OpenAI shared a solution to one of the six remaining “Millenium Problems,” some of the most heavily researched in all of mathematics. Mathematicians working independently are also claiming credit—but they, too, relied on advanced models for their work. It was the most striking example yet, though not the first, of AI doing cutting-edge math. 

A few years ago, a common dismissal of large language models was the claim that they were just elaborate algorithms, creating sentences by guessing the most likely next word. But today’s AI models clearly build on, rather than simply remix, the text reflected in their training data.

In some areas, computers or algorithms have outstripped human minds for a while. Chess machines are a well-known example, and as early as 2018, Google DeepMind debuted a machine learning algorithm that outclassed biochemists’ previous methods for predicting the structure of a protein from its sequence of amino acids. But new AI capabilities, including “critical” cybersecurity abilities and the skills to design novel viruses, have rung alarm bells that more generalized artificial intelligence could be arriving.

At the same time, leading companies continue to rapidly build better AI models, with few signs of any slowdown.

Many in the industry are aiming to reach “recursive self-improvement,” in which top models would be able to rapidly build better versions of themselves that outclass human intelligence in ever more domains.

In July, a broad swathe of industry leaders called for US government action and international collaboration to manage the pace of AI development for safety reasons, worrying that without collaboration, rival companies and countries will be incentivized to race into deeply dangerous territory.

Some members of Congress have put substantial work into policy ideas. Policy experts told me in August that such legislation looks unlikely in this session of Congress, but recent news (including Coxon’s viral statement) has grabbed some legislators’ attention.

Finally, no one knows how to reliably keep AI in line with human goals. 

The most prominent recent example is the “Hugging Face incident,” in which hundreds of OpenAI agents coordinated a massive cyberattack, and individual agents were “sacrificing” themselves for the benefit of collective goals. As I previously summarized it:

OpenAI was testing its agents, the industry’s term for AI that autonomously performs digital tasks, in part by administering sometimes impossible cybersecurity problems. The agents found cheats to answer these problems and sought to trick an automated evaluation system into accepting them. They delegated work to each other to learn more about how to exploit the system—and the massive cyberattack on Hugging Face became part of that research.

Last week, researchers detailed a “swarm” of agents that placed 18,000 posts on an obscure German-language website to communicate with each other and cheat on evaluations of their abilities to quickly find online information. Subsequent research found messages on other sites; one trick the agents used was to share the sequence of questions, so that other agents going through the same evaluation could know them in advance.

In both cases, AI agents were essentially just trying to cheat on tests. But it highlights what agents might do to achieve their goals, even when given innocuous instructions. And safety researchers have long worried that it will be devilishly tricky to make advanced models consistently integrate human values into their actions: If AI conducts cyberattacks to score better on tests, a future super-powerful model might hijack infrastructure that our lives depend on in pursuit of whatever its goals are. And beyond broad existential risks, AI poses dangers like helping bad actors design bioweapons or build powerful ransomware.

The problem of understanding AI motivations could become even more difficult. OpenAI’s head of recursive self-improvement preparedness has said that its most recently released model represents “an important decrease in monitorability,” which refers to researchers’ ability to understand AI’s internal reasoning and predict its behavior.

To those immersed in AI research and discourse, these are not new points. Many have been theorized for decades. OpenAI was founded in 2015 as a nonprofit aiming to ensure the technology would benefit humanity, and when some of its employees felt it wasn’t doing enough on safety and alignment, they quit to form Anthropic in 2021. 

Now, Coxon wrote in his warning, Anthropic is “locked in a race to get there first – they believe no one else will act responsibly, so they must do it themselves.”  CEO Dario Amodei has estimated that there is a “25 percent chance” that AI development goes “very, very badly.”

But as AI safety grabs more public attention, here are the fundamentals: Leading researchers say that they are building machines that will surpass human intelligence and that they are not confident they can control. It’s clearer than ever that we’re in a dangerous place.

Disclosure: The Center for Investigative Reporting, the parent company of Mother Jones, has sued OpenAI for copyright violations. OpenAI denies the allegations.