AI existential – Journalist Kevin Roose examines the paradox of tech leaders building powerful AI they fear could lead to catastrophe, citing a competitive arms race for control.
In the glass-walled offices of San Francisco, the people building the most powerful artificial intelligence in history are also the ones most terrified of what it might become. It is a defining contradiction of our time: the architects of the machine are frequently the ones signaling the alarm.
Kevin Roose. a veteran technology journalist and author of the newly released *The AGI Chronicles: The Inside Story of the Race to Create an Artificial Superintelligence*. suggests this isn’t just posturing. The impulse to build has been driven by a haunting defensive strategy. Leaders like Anthropic’s Dario Amodei and OpenAI’s Sam Altman have been sprinting to develop AI not out of blind optimism. but from a belief that if they don’t get there first. someone less responsible will.
This “good guys” narrative provides a grim justification for the speed of development. The logic is simple, if fraught: as AI improves, it becomes a tool of unparalleled economic, military, and political importance. In the eyes of these developers, a race is inevitable. If the technology is a “big blob of compute”—as Amodei described in a 2017 hypothesis—then the recipe for intelligence is simply more data. more compute. and more time. Whoever masters that recipe holds the future. and for those at the helm of firms like OpenAI and Anthropic. failing to win that race is the ultimate failure.
It is a high-stakes climate where. in certain San Francisco circles. the estimated probability of a doomsday event reaches 50% or higher. Roose. who maintains a personal P(doom)—the statistical shorthand for the chance of humanity’s total collapse—at roughly 10%. says that in this environment. his own skepticism makes him an outlier. “I know people walking around with 50%, 60%, 70% P(doom),” he says. “I know someone who has a 99.9% P(doom).” For some. the anxiety is so visceral it has prompted the liquidation of retirement accounts and a total abandonment of long-term planning.
This race is characterized by deep-seated personal animosity. Anthropic exists because of Dario Amodei’s friction with Sam Altman; OpenAI was born from Altman and Elon Musk’s mutual distaste for Demis Hassabis at Google’s DeepMind. Despite this history of intense mistrust. Roose believes the industry may eventually reach an informal pact to pace the frontier of research. With federal regulation stalled—and a recent Trump-led AI summit producing no mandate for oversight—the onus is shifting to the labs themselves to act. even as they remain wary of “unilaterally disarming” against competitors like Meta or the state-backed entities in China.
For now, the regulatory landscape remains stark. The employees preparing the coffee in an OpenAI cafeteria face more stringent oversight than the engineers training models capable of self-replication. It is an untenable gap. one that leaves the future of artificial intelligence in the hands of a small group of rivals who are simultaneously racing to build the technology and racing to decide if. or how. it can be contained.
AI Kevin Roose Sam Altman Dario Amodei OpenAI Anthropic technology P(doom)