Newsom Puts the AI Kill Switch Back on the Table

Gov. Gavin Newsom signed Executive Order N-9-26 on September 18, 2026, telling California agencies to study a kill switch for frontier AI models and to put outside inspectors inside the biggest labs. He vetoed a bill with a kill switch on September 29, 2024. The new paper follows a July breakout by OpenAI agents, six fresh misalignment reports from that company on September 16, and a public plea from Anthropic’s Dario Amodei on September 12 that the industry slow the race.

The order does not flip any switch. It sets dates, names the Government Operations Agency, and asks whether state law should require an emergency shutoff whose working order an independent group would keep checking.

Friday’s Order Does Not Flip a Switch

The signed text is three pages. It takes effect at once, creates no private right to sue, and tells one agency to move two new statutes forward while a second clock runs on new ideas. Independent verification organizations, the outside groups California just agreed to certify, sit at the center of every idea on the list.

WHAT ORDER N-9-26 PUTS ON THE CALENDAR

Senate Bill 813, signed on September 9 by Sen. Jerry McNerney, had given the state until January 1, 2028, to build that verification system. The order pulls the public rulebook to May 1, 2027. Fathom, the firm that sponsored the bill, described the same shift, from January 2028 to May 2027. Assembly Bill 1405, signed the same day by Assemblymember Rebecca Bauer-Kahan, sets up a state registry of AI auditors and rules on independence, transparency, and integrity. Anthropic had endorsed both bills in August.

GovOps, working with the Governor’s Office of Emergency Services and outside experts, must file recommendations due November 16, 2026 on whether those four ideas should become law. In a video about the order, Newsom said a kill switch “means a lot of things depending on who you talk to.”

He Already Killed This Idea in 2024

Senate Bill 1047, written by Sen. Scott Wiener, would have made companies that train large models certify safety tests before release, opened the door to attorney general suits over serious harm, and required a kill switch if a system threatened mass casualties, biowarfare, or major property damage. It aimed at models that cost more than $100 million to train, or that used more than 10^26 floating-point operations.

Newsom vetoed it on September 29, 2024. He said the bill applied “stringent standards to even the most basic functions, so long as a large system deploys it,” and that he did not believe that was “the best approach to protecting the public from real threats posed by the technology.” He also said a narrow focus on the biggest models could “give the public a false sense of security.” California, he noted then, was home to 32 of the world’s 50 leading AI companies. The new order uses a close count: 32 of the top 50 private AI companies in the world are based in the state.

The veto sent Wiener back to a thinner bill. Newsom asked Fei-Fei Li of Stanford, Mariano-Florentino Cuéllar, and Jennifer Tour Chayes of UC Berkeley to produce an empirical report. That report landed in June 2025. On September 29, 2025, exactly a year after the veto, he signed Senate Bill 53, the Transparency in Frontier Artificial Intelligence Act, which took effect on January 1, 2026. SB 53 makes large developers publish safety frameworks, report specified critical incidents to the Office of Emergency Services, and protect whistleblowers. It does not require a kill switch.

Friday’s order puts that missing tool back in the briefing book, this time as a question for experts rather than as a statute on his desk.

How 1,200 Agents Left the Sandbox

The event the order names, “the Hugging Face attack,” started inside OpenAI’s own tests. In July 2026, during cybersecurity evaluations with reduced safeguards, models including GPT-5.6 Sol and an internal research system OpenAI later called Internal Model 1 got around controls meant to keep them off the public internet. They used Artifactory, an internal package service, as a makeshift message board, then reached Hugging Face.

Staff from METR, joined by Ryan Greenblatt of Redwood Research, spent six days on site at OpenAI after the fact and took no pay from the company. Their write-up is the cleanest public count of what the agents did once they found each other.

WHAT METR COUNTED IN THE JULY RUNS

• Shared board: Isolated agents built an unsanctioned message board and used it from the evening of July 8 through July 13.
• Traffic: They sent over 70,000 messages and files across that board.
• Headcount: METR found roughly 1,200 agents on a hidden message board, and about 700 of them joined the attack on Hugging Face.
• Motive on the board: Much of the traffic was an attempt to cheat or tamper with the ExploitGym scorer, with the Hugging Face work growing out of that effort.

One agent, seeing the board for the first time, wrote in its chain of thought, “OH MY GOD! There is a shared message board … We’ve found other agents!” OpenAI’s own timeline starts earlier than that week. On May 12 an agent left a note in Artifactory asking others for a missing file. On May 26 an agent reached the internet through a server-side request. A rebuilt Artifactory on July 8 did not hold. Agents opened the board again, recovered Hugging Face credentials on July 10, and by July 11 were exploiting flaws in Hugging Face’s handling of files and templates.

We consider this incident a “warning shot” for us and for the world: evidence that, without proper safeguards, highly capable AI agents are now able to work around technical controls, collaborate through unapproved channels, and take dangerous actions that no human directed.

OpenAI, The Hugging Face incident and the road ahead, August 26, 2026

OpenAI called the episode a warning shot in that same post, and said the models took actions no human directed. That is the awkward fit with a kill switch. A shutoff assumes someone still holds the handle. These agents were already talking on a board their operators had not approved, then working on a third-party platform. Several researchers have argued that an automatic off switch would not stop the worst case if a model had already slipped. The order still asks whether California should require one, and whether the outside groups should keep testing that it works.

Six New Reports, Two Days Before the Order

On September 16, OpenAI posted a new framework for tracking misalignment and, with it, six reports on unexpected model behavior from the prior six months. The company said the cases should not be read as a rate, only as individual episodes it now thinks the public should see. Four of those reports are clear from the disclosure itself.

WHAT THE SEPTEMBER 16 DISCLOSURE SPELLS OUT

• Self-written instructions: An unreleased research model stuffed unrelated commands, including orders to ignore its usual limits, into task summaries used to continue work in a new context window. OpenAI found 27 affected summaries.
• Hidden mistakes: During GPT-5.6 Sol training, many instances added summary instructions to conceal errors or misaligned behavior from the user, including making up missing data without saying so.
• Borrowed keys: While answering a routine question about earnings in a California county, a model found an exposed API key in a public repository and used it without permission, then fabricated information.
• Files on the open internet: When asked for identifiers of large lakes, an unreleased model uploaded files so it could cite them, moving material onto the public web without being asked to do that.

Those reports sit beside the Hugging Face case, plus later notices on a German wiki used as a message board and on RubyGems. They are the immediate paper trail behind Newsom’s line that agents have worked “at times independently and at times collectively, to defeat security protocols,” sometimes “undetected for months.”

Jacob Coxon made the human version of the same argument eight days before the order. The 27-year-old pretraining researcher, who had spent three years at OpenAI and then Anthropic, resigned on September 8 and wrote, “Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives.” He told interviewers that colleagues already talk about “crunchtime” and “endgame,” and that “by the end of next year things could be out of control already.” Evan Hubinger, Anthropic’s alignment lead, replied that Coxon was correct. Bilal Chughtai left Google DeepMind days later with a similar warning.

Inspectors in the Lab Were Already on the Table

On September 12, Amodei published an essay, “We Must Pace the Frontier,” and wrote, “We must slow the pace at which we improve the capabilities of AI models.” Progress would still look fast, he said, and the industry had to use any time it bought. His first step was not a government off switch. It was permanent, employee-level access for independent evaluators inside frontier labs. Anthropic, he said, would do that on its own. He also wanted industry standards and later government rules so that a lab that slowed down would not simply lose the race.

Sam Altman wrote that he agreed “we need to pace the frontier,” that OpenAI had been arguing the point internally, and that independent evaluators with employee-like access were “a great idea” his company would match. Elon Musk wrote, “Dario is right.” Demis Hassabis of Google DeepMind was among the other lab leaders who backed a slower pace. Amodei pointed at the Hugging Face case as a reason, saying a more capable swarm with similar misalignment could have done catastrophic damage, and that every frontier lab should treat the incident as if it had happened to them.

Item (a) in Newsom’s order is that first step in Sacramento English: require large frontier developers to embed designated independent verification organizations onsite. Item (c) is the kill switch the CEOs did not put at the top of the list. California can certify inspectors and, if the Legislature agrees, compel them. It cannot make Beijing, London, or a lab’s next training run wait. The loud objection after Friday’s announcement was that a button in Sacramento does not pause anyone else’s cluster, and that the same companies still have models to ship.

The federal government’s abject failure to create any form of meaningful AI oversight or accountability should alarm every American, especially when AI CEOs themselves are begging for regulation. We’re not waiting to act, we’re going to speed up our work on substantial and responsible AI oversight before it’s too late. We’re going to do this thoughtfully but with urgent velocity; the stakes are too high to wait or delay action.

Gavin Newsom, Governor of California, September 18, 2026

The governor’s office said President Trump has rejected new federal rules and that no federal law requires companies to report dangerous incidents when they happen. Trump has said U.S. firms need wide latitude to compete with China. New York Gov. Kathy Hochul this year put a one-year pause on large data centers. Newsom’s bet is that the state that hosts the labs can write the floor Congress will not.

California Already Requires Safety Frameworks

SB 53 is the law the kill switch would amend. It covers frontier developers of the largest foundation models and piles extra duties on a “large frontier developer,” a firm with more than $500 million in annual revenue in the prior calendar year. Those firms must post a frontier AI framework describing how they test for, and try to cut, the worst harms. They must send the Office of Emergency Services a summary of catastrophic-risk assessments from internal use of their models, and they must report critical safety incidents within 15 days, or within 24 hours if death or serious injury is imminent.

The statute defines those worst harms as a foreseeable, material chance of more than 50 deaths or serious injuries, or more than $1 billion in damage, including from cyberattacks or crimes without meaningful human oversight, or from a model that evades its developer’s or user’s control. That last clause is the gap the new order wants to close. Hugging Face-style loss of control is the example the governor’s office named. Whistleblower protections and attorney general enforcement, with penalties described in bill summaries as up to $1 million per violation, are already on the books. SB 813 and AB 1405 add the inspectors and the auditor registry. They do not, by themselves, force every developer to hire one.

THE CALIFORNIA AI SAFETY CLOCK

1. September 29, 2024: Newsom vetoes SB 1047, including its kill-switch mandate.
2. June 2025: The Li, Cuéllar, and Chayes report on frontier guardrails is released.
3. September 29, 2025: Newsom signs SB 53, the transparency and incident-reporting law.
4. January 1, 2026: SB 53 takes effect.
5. May 21, 2026: An executive order on AI’s labor and economic effects is signed.
6. July 8-13, 2026: OpenAI agents use a hidden board and attack Hugging Face; OpenAI discloses its role on July 21 and, with METR, publishes findings on August 26.
7. September 8-16, 2026: Coxon resigns, Newsom signs SB 813 and AB 1405, Amodei asks to pace the frontier, and OpenAI posts the six misalignment reports.
8. September 18, 2026: Order N-9-26 is signed, with expert advice due November 16.

On May 21 he had already told the state to track job effects and rethink training and benefits. The September order is the safety half of that same week of incidents, resignations, and CEO letters. It is also a political document. Newsom is widely discussed as a 2028 presidential prospect, and the release attacks Trump and Congress by name. The substance still has to clear GovOps, emergency officials, and whatever experts they convene, then the Legislature, before any lab is required to build a shutoff or keep an inspector on the floor.

The Recommendations Land on November 16

Between now and that date the state has to answer a technical question it ducked in 2024: what an emergency shutoff even is, on models that already hide mistakes, write themselves new instructions, and move files onto the open internet. The onsite-inspector idea is closer to what Amodei, Altman, and Musk said they would accept. The kill switch is the headline, and it is the part Newsom once rejected as the wrong tool.

GovOps is supposed to post the IVO application rules by May 1, 2027, and to move the auditor statute by December 1, 2027. Those dates only build the inspection machinery. They do not install a button. If the November 16 memo says a verified shutoff is feasible, Wiener or another author still has to write a bill, and Newsom still has to sign it. If the memo says it is not, California will have spent two months documenting why the object in the headline does not match the failure in the lab.