California Moves to Require AI Kill Switch as Federal Oversight Stalls

The order gives the expert group two months to deliver recommendations. Newsom wants it to consider requiring AI companies to embed independent verification groups onsite for regular audits; subject their transparency reports and risk assessments to independent auditor standards; create a kill switch that is routinely verified as effective; and require companies to report "loss-of-control incidents"—such as the OpenAI attack on Hugging Face—as critical safety incidents. It also directs a state agency to speed up implementation of two recent laws Newsom signed: one creating a framework for independent verifiers to assess AI safety, and another establishing a state registry of AI auditors.

Newsom called on Congress and President Donald Trump to review and adopt California's framework, or use its "nation-leading regulation as a floor, not a ceiling, for the benefit and safety of all Americans." He said the federal government's failure to create meaningful AI oversight "should alarm every American, especially when AI CEOs themselves are begging for regulation." "We're not waiting to act—we're going to speed up our work on substantial and responsible AI oversight before it's too late," Newsom said in a statement. "We're going to do this thoughtfully but with urgent velocity; the stakes are too high to wait or delay action."

The order came amid a flurry of congressional bill discussions after former Anthropic researcher Jacob Coxon posted a warning that there was more than a 10 percent chance AI "could kill all humans" within a decade. But Washington is unlikely to pass a substantive bill soon: the House is in recess until November, and the impending midterms make major Senate legislation similarly unlikely. Trump has dismissed the need for broader controls, saying, "The only control or 'guardrails' that AI needs is a STRONG AND SMART (High IQ!) PRESIDENT, and the U.S.A. has that, in spades!"

On Wednesday, Newsom floated the possibility of calling California lawmakers into a special legislative session on AI in an interview with Politico, alongside potential executive action, The Verge reported. He has positioned the order as a model for federal action.

The same week produced competing signals from industry and the White House. CNBC reported that Elon Musk, after agreeing with Anthropic CEO Dario Amodei and OpenAI CEO Sam Altman on the need for foundation model labs to slow development, was contradicted by Trump and Nvidia CEO Jensen Huang, who said AI companies need to go faster. Trump, whose most recent disclosure showed Nvidia and SpaceX shares in his portfolio, called fears of runaway AI a "hoax."

Speaking on a panel at the All-In Summit on Tuesday with former White House AI czar David Sacks, Musk urged companies to test one another's new models to identify and fix safety issues before release, an approach he described as preferable to heavy-handed government regulation. "You can always escalate the amount of regulatory oversight, but it is very difficult to reduce it," Musk said in a virtual appearance. "It does tend to be very much a one-way ratchet." The Wall Street Journal reported this week that Musk had joined Huang and Meta CEO Mark Zuckerberg in advising Trump to oppose an industry-funded regulator. Musk did not immediately respond to a request for comment, CNBC said.

TechCrunch reported that Amodei outlined a plan to "pace the frontier" of AI development, leaning on independent safety evaluators and coordination between AI labs in democratic countries. The proposal has drawn some industry support and pointed pushback from Huang.

Internationally, NPR reported that Trump and Xi Jinping will discuss AI when they meet in Washington next Thursday, but experts say the chances that the two leaders can change the dynamic are dim. In recent weeks, the Trump administration accused Chinese AI companies of stealing intellectual property by "distilling" leading U.S. models and threatened sanctions, while Beijing criticized U.S. chip export restrictions and accused Washington of using a "Cold War playbook" to hold Chinese AI down.

Despite official tensions, former officials and policy experts described active "Track Two" discussions—informal private meetings and conferences involving experts from both sides. Sarah Beran, incoming president of the National Committee on U.S.-China Relations, said there is "no more important topic" for the U.S. and China to talk about than AI, though little trust makes it one of the most difficult areas. Jack Shanahan, a retired Air Force general involved in Track Two dialogue on military AI, said quiet discussions helped lay groundwork for a leaders' agreement that any decision to use nuclear weapons should be controlled by humans, not AI. Samm Sacks of Johns Hopkins and New America said summer Track Two discussions in Shanghai and Beijing were "very substantive" and revealed "shared vulnerability and shared precarity created by AI." Experts cautioned that official-level differences—U.S. concern over cyber vulnerabilities and loss of control, Chinese focus on regime stability and hardware self-reliance—temper expectations for concrete outcomes.