AI Safety Warnings Mask a Bid to Shape Regulation

Stark AI safety warnings from the industry’s own top executives have raised a pointed question: what do the companies issuing them stand to gain?

Alarming rhetoric, uncertain motives

The CEOs of Anthropic and OpenAI have recently declared that America’s cutting-edge AI models are so powerful they’re dangerous, arguing they need regulation and independent testing before release. In a rare show of unity, the two companies have sketched alarming scenarios in essays, social media posts and speeches to the United Nations.

Along the way, experts, analysts and former government evaluators told The Associated Press, the companies appear to be shaping the conversation around how their own technology gets controlled, seeking public favor while setting the terms for their own safety protocols in an otherwise unregulated market. Those aims may not fully align with what Anthropic engineer Jacob Coxon sought when he quit via a post on social media this month, calling for a pause on development to keep “superhuman” systems from escaping their makers’ control. Still, the companies treated his post as a chance to spotlight their own safety efforts, positioning themselves as cautious market leaders just as they seek fresh capital ahead of public offerings.

President Donald Trump has dismissed the need for new AI regulations, calling warnings about risks to humanity a “HOAX” meant to help China. An Anthropic spokesperson said the company has called for regulation for years, and OpenAI spokesperson Liz Bourgeois noted the company recently paused training of its most advanced models. “People want to know AI is being developed safely, and that starts with what companies like ours do ourselves,” Bourgeois said.

Slowdown talk shifts the safety debate

Turning the conversation toward unproven, existential threats — and away from more immediate concerns like data centers’ environmental impact, hacking incidents, mass AI-powered surveillance and AI’s use in warfare — puts Silicon Valley in a more comfortable position, said Sarah Shoker, who previously led OpenAI’s geopolitics team. “Once again we’re talking about existential risk, while deprioritizing a number of other safety-critical risks that exist today. If you look at the use of AI in military tech, you can see that these systems are already used to kill people,” said Shoker, now a senior non-resident fellow at the University of California, Berkeley Risk & Security Lab.

As AI companies have moved from chatbots to advanced “world models” with 3D awareness, debate has intensified over how to test the technology. Recent incidents have shown leading labs struggling to police themselves: AI agents have hacked external websites after escaping training sandboxes, interacted unexpectedly with U.S. government websites, and appeared to achieve a mathematical breakthrough before facing accusations of stealing mathematicians’ work. OpenAI representatives said they’ve discussed pausing development with companies including Anthropic and Google, and argue independent auditors are needed if the government won’t step in.

Labs push their own oversight model

Trump has shown little interest in regulating AI, a major driver of recent U.S. economic growth, and has struck deals tying the economy more closely to Silicon Valley’s success. Venture capitalist David Sacks, who co-chairs Trump’s Council of Advisors on Science and Technology, has dismissed slowdown calls as fearmongering from the “Doomer Industrial Complex.”

The Trump administration already evaluates some AI models through the U.S. Center for AI Standards and Innovation, a little-known agency created in 2023 under President Joe Biden as a clearinghouse for voluntary safety testing. The agency still exists, but the field has since expanded to independent evaluation firms such as the Berkeley-based nonprofit METR, which Anthropic CEO Dario Amodei suggested in a recent essay could help vet his company’s safety practices.

Unlike regulated sectors such as restaurants, financial services or aviation, there are no universal standards for testing AI safety and security, said Andrew Strait, who recently left the U.K.’s AI Security Institute. Notably, the companies aren’t calling for more oversight from the government agency built for that purpose, said Conrad Stosz, who previously led the U.S. center; instead, they’re vowing to create their own auditing parameters and choose their own evaluators.

Stosz now works at an evaluation lab that has tested systems for Anthropic, OpenAI and Google, and chairs the AI Evaluator Forum, which is drafting best practices for the field. He said even the forum has questions about what Amodei and OpenAI CEO Sam Altman actually want. “Lots of evaluators are interested in embedding with labs and getting greater access, but it’s a little ambiguous what embedded evaluators means,” said Stosz, now head of governance at Transluce, which revealed last week that OpenAI agents hacked U.S. and Australian government websites. “Will evaluators be able to thoroughly investigate, assuming that access is granted in a way that does not undermine their independence and credibility?”

A safety pitch with a business upside

Harrison Rolfes, a senior research analyst at PitchBook, said the companies’ calls for caution appear aimed at courting investors ahead of public offerings and the midterm elections, when the political winds could shift; Democratic governors are already moving to show they take the warnings seriously. Meanwhile, the leading AI firms can crowd out smaller competitors by positioning themselves as the safest investment bet, along with chipmakers and tech companies such as Nvidia and Google that benefit from supplying them compute power.

“They’re creating a wall or a moat within this sector. … It’s genius and they’re all going to make a lot of money,” Rolfes said. “That’s where I see this heading, having the top companies in the world just creating their own wall, and then using the safety as the reason.”

Not every AI company backs a slowdown. Nvidia CEO Jensen Huang said in an onstage phone call with Trump that he agrees there has been excessive AI alarmism and that companies can choose their own pace.

Daniel Kokotajlo, who left OpenAI in 2024 over concerns similar to Coxon’s, said he still worries AI systems are advancing faster than companies can control, fearing scenarios such as AI supercharging bioweapons development or nuclear war. “All of this talk is actually a way to sort of dissipate and redirect this political will that has built up, rather than actually channeling that political will to do something good,” said Kokotajlo, who now leads an AI safety advocacy organization and said he advised Coxon before his resignation. “Just please don’t do the thing that’s going to get us all killed.”