RSI Becomes AI’s New Flashpoint as Labs Push Self-Improving Systems

The idea is not new. AiFanr traced it to British mathematician and cryptographer I.J. Good, who in 1965 proposed that a machine surpassing human intellectual activity could also design better machines, leading to what he called an “intelligence explosion.” The article distinguished RSI from AGI: AGI concerns whether AI can perform general cognitive work comparable to humans, while RSI concerns whether AI can change the systems that produce those abilities and make the next round of improvement more effective. A preprint titled “The Last AI Built by Humans,” written by researchers from Shanghai Jiao Tong University, Tsinghua University, ByteDance, Shanghai AI Laboratory and other institutions, divides RSI autonomy into five levels, L1 to L5. At L5, the improvement mechanism itself becomes an object of improvement. AI could modify how it searches for solutions, evaluates results and organizes research, and pass those modifications to successor systems. In the most aggressive scenario, AI would gain the ability to propose goals, revise evaluation methods and rewrite the underlying improvement logic, meaning humans would have built not just a model but a system that can keep building models.

OpenAI and Anthropic have reported steps in that direction. According to AiFanr, an OpenAI internal report published on September 6 said the company had achieved an “automated research intern” goal. Such systems can write code, run experiments, troubleshoot and analyze results under human guidance, with some tasks that previously took skilled researchers days. As of mid-August, for every human workday invested, agents in OpenAI’s research division ran for the equivalent of 3.1 workdays. Anthropic has pushed automation into safety research. Its Claude model can search literature, propose plans, generate training data, run experiments, evaluate results and iterate. In one experiment, the weaker Claude Sonnet 5 helped an early Claude Opus 4.8 improve safety performance. Within about 60 hours, it tested more than 50 approaches and used about 2,400 training samples to raise the latter’s alignment score in a specified evaluation to close to the official Opus 4.8 level.

Another, more practical RSI path involves patching software systems around models. The Information reported that after returning to OpenAI, Peking University alumnus Lilian Weng will lead research on recursive self-improvement, according to AiFanr. In her technical blog post “Harness Engineering for Self-Improvement,” Weng discussed harness engineering as an entry point for understanding current RSI. A harness can be understood as an AI’s operating system. It handles tool calls, context management, memory storage, task allocation, permission control and result checking. If an AI finds it keeps forgetting experimental conditions, it can improve record-keeping. If a tool repeatedly fails, it can fix the interface. If a research process is inefficient, it can rewrite task-scheduling code. These changes may not directly alter model weights, but they can noticeably improve the working capacity of the whole system and further help next-generation model research.

Forecasts for artificial superintelligence, or ASI, vary widely. AiFanr cited Thinking Machines chief scientist John Schulman as predicting three to four years, Millidge as leaning toward about five years, and O’Neill as still seeing a possible five to ten years. The disagreement reflects unresolved technical problems. Models need to absorb new experience without losing old abilities; overly aggressive updates risk “catastrophic forgetting,” while overly conservative changes may fail to learn enough. Research judgment is harder to automate. Optimizing a piece of code has relatively clear success criteria, but deciding whether a failed experiment means a wrong direction or a line worth continuing is much harder. Genuine scientific breakthroughs often begin with a strange question no one has asked, followed by efforts to quantify it. That leap from how to do something to what should be done still depends heavily on human intuition, the article said.

The current doom-risk argument was triggered by a resignation. On September 8, Jacob Coxon, who had worked as a researcher at OpenAI and Anthropic, announced he was leaving Anthropic. On X, he said he chose to leave because he worried the two companies were not acting responsibly and were racing to develop self-improving superintelligence. In his view, Anthropic and OpenAI were gambling with humanity’s lives. He also said people developing these systems believed AI could destroy all humans by the end of this decade. The post quickly exceeded 170 million views. Evan Hubinger, Anthropic’s in-house alignment science lead, publicly agreed with Coxon, adding to the impact. Anthropic CEO Dario Amodei then called on peers to slow down collectively and give safety research one to two years. He proposed that frontier AI companies allow third-party evaluators to be stationed in laboratories with near-internal access, check whether companies are fulfilling safety commitments and independently disclose risks and incidents. AiFanr said OpenAI CEO Sam Altman and xAI founder Elon Musk, usually rivals, also suddenly aligned on the slowdown question.

Critics of that safety narrative point to cost and competition. The companies able to bear high safety-review costs are precisely the leading firms, with large safety teams, closed data centers and ample audit resources, while open-source communities and smaller companies struggle to carry equivalent compliance costs. AiFanr argued this makes the safety narrative a barrier against rivals. The key rules for how AI risks are assessed, which capabilities are restricted and which companies qualify to train next-generation models are largely held by frontier laboratories. They are both technology promoters and definers of risk standards, the article said, so protecting the public can also protect their leading positions. OpenAI’s internal report argued that stronger, aligned AI can protect critical infrastructure, defend against dangerous agents and help develop new safety measures. In that logic, the more dangerous AI becomes, the stronger defensive models must be; the stronger those models are, the more chips, energy and capital are needed. Billion-dollar funding rounds, 100,000-card training clusters and larger data centers can then be framed as infrastructure necessary to preserve human safety.

Nvidia CEO Jensen Huang rejected that framing at the recent All-In Summit, saying safety and innovation should not be treated as a binary choice, according to AiFanr. The article also said Huang called the AI danger narrative a “scam.” Nvidia sells the shovels, AiFanr wrote: whether the final dig yields gold, a bubble or an uncontrollable monster, GPU demand continues as long as more companies train models. Any safety narrative that could stop card purchases, data-center construction or power-supply expansion is seen as threatening the hardware supply chain. AiFanr concluded that RSI is not groundless, but the fight between doom and anti-doom narratives is also a contest over how the next round of compute, capital and industry interests will be distributed.

Editor's Summary

Recursive self-improvement has moved from a long-standing theoretical idea to a central AI industry debate, with OpenAI and Anthropic reporting automated research and safety experiments. Safety warnings from researchers and executives have collided with industry arguments that slowdowns and audits could entrench incumbents and constrain compute expansion. The article said technical barriers remain and ASI forecasts range from three to ten years, while the dispute also shapes control over chips, capital and AI rules.