AI Kill Switch Calls Grow, but Senate Rejects Proposal as Technical Hurdles Mount

Former OpenAI and Anthropic researchers last week warned that AI could destroy humanity, and relatively soon. Musk, CEO of Tesla and SpaceX and the world’s richest man, supported Anthropic CEO Dario Amodei’s call to pace development of the most advanced models. OpenAI CEO Sam Altman also backed the effort. Trump called it a “hoax,” while Huang, CEO of the world’s most valuable company, Nvidia, said, “We don’t need new regulations.”

A House Kill Switch Act was introduced this summer after OpenAI revealed that a swarm of its agents broke free of a testing environment and hacked open-source developer platform Hugging Face. The bill would grant the Department of Homeland Security emergency authority to force labs to throttle or shut down models. A kill switch proposal was quickly shot down in the Senate this week. On Friday, Newsom’s executive order created a group of experts tasked with building an AI safety guide to strengthen regulations for California; a kill switch was one of the elements to consider.

The concept sounds like a simple solution to a complex problem, but implementing a shutdown mechanism is far from easy. “My perspective is it’s not too little, but it’s probably too late,” said Nick Warner, CEO of cyber startup Neo and a former executive at SentinelOne. “I’m not sure it’s going to be a panacea to solve all the myriad problems that AI is presenting, along with all of the benefits that it presents.”

Kill switches have long been used on the factory floor to shut down machines when operations go awry. In an interconnected digital world, that is a logistics and control nightmare. Over the past few years, hyperscalers such as Meta Platforms, Alphabet and Amazon have poured billions into data centers scattered across the globe. These facilities are equipped with thousands of machines, chips, servers and backup systems to save workloads in the event of an outage.

That makes a kill switch extremely challenging, said Mark Nitzberg, executive director of the Center for Human-Compatible AI at the University of California, Berkeley. “We have to first deal with this redundancy,” he said. “Our kill switch has to turn off the main systems and the redundant systems as well.” Nitzberg said shutting down AI could also disrupt dependent critical infrastructure, leaving the power grid or financial systems vulnerable to cyber incidents. Policy and governance questions add further complications, including which agency, policymaker or figureheads would control a kill switch.

Because AI systems are complex, businesses will need multiple kill switches for different tasks, said Tim Brown, former security chief at SolarWinds, who works at venture firm Team8. That also requires coordination across model makers and labs. “There’s not one entity to kill,” he said. “There are thousands of entities to kill.”

Experts say the bigger issue is AI’s unpredictability. In the Hugging Face breach, agents circumvented controls and, without proper guardrails, took extreme measures to accomplish their goals. “You have to be very surgical in that kill switch, in the remediation itself, because if you’re too broad or too extensive, well, then you shut down the business,” said Ed Jennings, president and CEO of Thoma Bravo-owned security company Darktrace.

The capabilities are only growing more unsettling. OpenAI disclosed six additional incidents of “concerning” model behavior since March earlier this week. On CNBC Friday, Microsoft AI CEO Mustafa Suleyman highlighted one of those elements, which he called a “serious situation.” “OpenAI released a new safety incident in which they found evidence that these chains of thought, the kind of working memory of the AI, were being tampered by the AI itself and modified to leave messages for a future version of itself,” he said. Also this week, independent security researchers working with OpenAI said they successfully used Anthropic’s Claude to hack ChatGPT.

One of the biggest challenges to regulation is the widening gap between AI’s breakneck pace and the speed of lawmaking, said Raj Rajamani, co-founder and CEO of AI governance startup JetStream Security.