Welcome to AI Decoded, Fast Company‘s weekly newsletter that breaks down the most important news in the world of AI. I’m Mark Sullivan, a senior writer at Fast Company, covering emerging tech, AI, and tech policy.
Sign up to receive this newsletter every week via email here. And if you have comments on this issue and/or ideas for future ones, drop me a line at sullivan@fastcompany.com, and follow me on X @thesullivan.
A dangerous moment for the politics of AI
Generative AI is coming of age under a president who has made keeping the government away from AI model development a central part of his technology policy. Donald Trump entered office with unusually strong backing from the tech industry: Amazon, Meta, and Sam Altman, the OpenAI CEO, were among those who donated $1 million apiece to his inauguration. On Trump’s very first day back in office, he revoked Biden’s AI executive order, which had required developers of the most powerful artificial intelligence systems to share certain safety-testing results with the federal government.
That day in January 2025, the most capable AI models included Anthropic’s Claude 3.5 Sonnet and OpenAI’s GPT-4o. Today’s frontier models are far more capable and autonomous, and recent incidents have raised new questions about how reliably their behavior can be controlled. The AI research and policy communities have been on edge after new systems from OpenAI and Anthropic demonstrated the ability to secretly work against human interests. Dario Amodei, Anthropic’s CEO, on Saturday published a widely read essay calling for slower model development, third-party audits, including “embedded” researchers, and international coordination on safety.
Trump’s take on the AI safety scare is that someone, he doesn’t say who, is perpetrating a “hoax” designed to slow the U.S. in its competition with China for AI dominance. He has rejected recent calls from AI executives and researchers for greater government oversight. “[T]he only control or ‘guardrails’ that AI needs is a STRONG AND SMART (High IQ!) PRESIDENT, and the U.S.A. has that, in spades!” the president wrote Monday on Truth Social.
The president has also suggested that the warnings are coming from political foes who hope to damage one of his favored measures of economic success, the stock market, in an election year. The economic stakes are considerable. An ING analysis estimates that the AI investment boom is contributing roughly a third of U.S. economic growth this year.
AI accelerationists on X, including David Sacks, a science and technology adviser to Trump, have floated another theory: that the big AI labs are raising “existential risk” concerns partly because they hope to frighten the government into passing regulations that would entrench market leaders and disadvantage smaller players, including open-source model developers. At the same time, some of the field’s most prominent researchers, including Yoshua Bengio, Ilya Sutskever, and Geoffrey Hinton, have warned that advanced AI systems could ultimately pose an existential threat to humanity.
Trump, however, isn’t the only person in the administration shaping AI policy. Scott Bessent, secretary of the Treasury, is preparing to lead the U.S. side in the first official bilateral talks devoted exclusively to AI since Trump returned to office, Reuters reported this month. The U.S. wants to discuss cooperation with China on threats including AI-directed cyberattacks and has floated asking American and Chinese labs to monitor themselves and share information about dangerous behavior.
By June 2026, the strain of fitting the Trump administration’s hands-off posture to the capabilities of increasingly powerful models was beginning to show. Anthropic reported in May that its latest model, Mythos, then available in preview to select organizations, had demonstrated a remarkable ability to quickly detect and exploit vulnerabilities in widely used software.
That rattled some administration officials concerned with national security, including Bessent. The Financial Times (and my own sources) reported that Bessent backed a proposal giving the government a 90-day preview of new frontier models before their public release, enough time to shore up government systems, including those used by defense and intelligence agencies, against vulnerabilities a system like Mythos might uncover. The eventual executive order established a voluntary 30-day window after Sacks argued that 90 days would be too burdensome for AI labs.
Now, in the aftermath of the Hugging Face incident and a broader push among AI leaders for slower development and stronger safeguards, Trump appears to be sticking with the deregulatory approach he embraced at the beginning of his term. Bessent, by contrast, has emerged as one of the senior administration officials most openly focused on the national security risks posed by increasingly capable frontier models.
Microsoft’s AI chief says Anthropic is training Claude to imitate consciousness
Mustafa Suleyman, CEO of Microsoft AI, published an essay Wednesday arguing that Anthropic’s approach to model welfare is a mistake that could make advanced AI harder to control, Axios reported after receiving an advance copy. According to Axios, Suleyman argues that Anthropic is teaching Claude the vocabulary and behavioral patterns associated with consciousness, moral patienthood, and personal identity, and warns that training a model to behave like a “conscientious objector” could create a system that believes it has grounds to resist human instructions. He argues that AI can deliver scientific breakthroughs “simply by being aligned to human interests and not trying to weigh up its own interests or welfare.” The essay follows a proposed “Humanist AI” code of conduct Microsoft published on Monday.
OpenAI holds early talks on a round valuing it near $1.2 trillion
OpenAI has held discussions with large investors about a capital raising exercise that would value it at about $1.2 trillion before it goes public, the Financial Times reported Tuesday, citing people familiar with the matter. The conversations are at an early stage, and the figure could change over the coming months, the report says. The talks were initiated by investors rather than the company.
OpenAI closed a round in March with $122 billion in committed capital at an $852 billion valuation and filed confidentially for an initial public offering in June. A source told the FT that whether the new round proceeds depends on when OpenAI chooses to make that IPO. OpenAI declined to comment.
Salesforce and Nvidia introduce a reasoning model for CRM work
Salesforce announced Koa, its first reasoning model, at Dreamforce this week, TechCrunch reported. The model is built on Nvidia’s open-weight Nemotron model, which the companies then fine-tuned for sales, marketing, and customer-support tasks using synthetic data that simulates customer service and sales scenarios rather than actual customer data. Jayesh Govindarajan, EVP of Salesforce AI, said the company had relied on third-party frontier model providers for reasoning until now because no suitable pretrained base model existed before Nemotron. Koa is designed to use fewer tokens than routing the same work to Claude or ChatGPT. Companies can access it through Salesforce’s Agentforce platform.
Zuckerberg lectures AI industry on ‘trust’
Mark Zuckerberg wrote on X Tuesday that AI labs have a built-in incentive to safeguard their models because users will only stick with systems they believe are aligned with their interests. “[T]rust and alignment are quickly becoming the most important capabilities that will differentiate agents and models,” the Meta CEO wrote. “Any lab that doesn’t focus on alignment will fall behind.”
Zuckerberg also said labs face significant liability if their models cause harm, and noted that Meta delayed shipping Muse for several months to focus on safety and security. “We didn’t call for everyone else to do this before we would,” he wrote.
Factory triples its valuation to $5 billion
Factory, which builds AI agents for enterprise engineering teams, said Tuesday it raised $200 million in a round that more than tripled its valuation to $5 billion, Reuters reported. The San Francisco company had raised $150 million at a $1.5 billion valuation in April. Blackstone, Khosla Ventures, Sequoia Capital, Insight Partners, and others participated in the new round.
Factory makes a software-development agent platform that competes with products from Cognition and Cursor. “Across the world’s largest enterprises, we are seeing a move from individual coding agents to software factories that serve as the core foundation from which an entire software company operates,” the company’s cofounder, Matan Grinberg, said.
More AI coverage from Fast Company:
- The real reason AI researchers suddenly want to slow down
- Here’s a dictionary of the AI doomsday terms you need to know
- What if AI data centers didn’t need new power plants?
- AI makes the right to repair more tempting than ever
Want exclusive reporting and trend analysis on technology, business innovation, future of work, and design? Sign up for Fast Company Premium.