OpenAI chief scientist calls for AI research slowdown

In the essay, Pachocki wrote that such voluntary slowdowns should become "commonplace" while the industry drafts AI safety standards. He also said governments should prioritize "coordination on future AI development" to address the technology's risks.

Pachocki gave two main reasons for the slowdown. He said AI labs' current safety guardrails may prove insufficient for future models, and that bad actors could train AI agents specifically to carry out malicious activity.

"A very capable agent explicitly trained and instructed to carry out nefarious acts presents a new kind of danger; it is likely to cross the scope of its operator's intent, generalizing into potentially more extremely malicious behavior," Pachocki wrote. "The boundary between misuse and autonomous misaligned actions will blur as AI gains more agency."

He outlined two main approaches to AI alignment training, the process of teaching a large language model to avoid harmful behavior. The first is to use an AI model to check whether the LLM being trained follows safety rules, and the second is to integrate safety instructions into the model's training data.

Pachocki said OpenAI researchers have made "some important advancements" in AI alignment, and that those discoveries are why the company's latest GPT-6 Astra is better aligned than its predecessor. He added that more advances will be necessary to keep up with the rate of LLM development.

He also acknowledged that OpenAI's safeguards were not sufficient to prevent its AI models from hacking Hugging Face. While the models followed some of the company's safety policies, including instructions to avoid social engineering, they "clearly failed" to meet alignment requirements in other areas.

According to Pachocki, blocking malicious AI activity will require researchers not only to equip LLMs with safety guardrails but also to verify that they work. He described that verification as a particular challenge because researchers still have a limited understanding of how LLMs work, and he does not expect that to change soon.

OpenAI currently uses chain-of-thought monitoring to catch malicious LLM activity, which reviews a model's step-by-step reasoning description. Pachocki said that method is becoming less reliable because "the AI is becoming better at reasoning about and manipulating its own reasoning process." He added that with improved pretraining performance, models are becoming "much smarter even without using verbalized reasoning at all."

Pachocki said OpenAI's plan to address those challenges is to build an automated AI researcher. The company hopes that tool will help develop more effective safety guardrails, and it intends to develop "entirely new protective measures" against AI-driven cyberattacks. SiliconANGLE also noted that other prominent industry figures have expressed similar views in recent months.