Major artificial intelligence (AI) developers are calling for a slowdown, arguing that AI models are heading toward a stage where they evolve on their own without human intervention. They cite the risk that AI is advancing so rapidly—entering an era of agents that reason and act independently—that if its capabilities reach a level humans cannot control, it could bring disaster to society.
In a blog post Thursday, Anthropic proposed an international agreement that would include the option to slow down or temporarily halt development, warning that failure to control AI's recursive self-improvement (RSI) could cause enormous chaos in society. "AI that builds itself could bring benefits to various fields such as science and medicine, but it could also raise the risk of humans losing control over AI systems," Anthropic warned.
AI recursive self-improvement refers to AI fully autonomously designing and developing its own successor models. When chatbots first appeared, humans led every step, and current agent AI still finds and solves tasks on its own within a human-designed framework. By contrast, recursive self-improvement is the stage in which AI itself acquires the code and algorithms needed for successor models and designs and develops them. As interest in AI recursive self-improvement has grown, the International Conference on Learning Representations (ICLR), one of the top conferences in deep learning, held its first workshop on the topic in April.
Claude, developed by Anthropic, is already evolving on its own in some functions. In March 2024, the Claude Opus 3 model handled software tasks that would take a person four minutes, but a year later, Sonnet 3.7 handled tasks lasting an hour and a half. Another year later, the Opus 4.6 model that emerged advanced to the point of instantly processing work that had taken 12 hours. "At this rate, next year it will perform even tasks lasting several weeks," Anthropic predicted.
OpenAI, Anthropic's biggest competitor, also stressed in a blog post last December that it is "researching how to safely develop and deploy AI capable of recursive self-improvement," adding, "We want systems to consistently follow human intent and not engage in catastrophic behavior."
Mostafa Dehghani, a researcher at Google DeepMind, said through India's platform Office Chai that "recursive self-improvement is quietly happening at almost every major AI lab," noting that "while it is not yet fully automated, it is clearly heading toward full automation." He pointed out that most people are unaware of AI's self-improvement because, just a few years ago, it was merely theoretical.
However, views on AI's self-improvement are divided. Unlike Anthropic, which places the greatest emphasis on ethics in the AI ecosystem, Meta takes the position that it also holds high utility value. Meta CEO Mark Zuckerberg warned in a policy document on the future of AI released last year, "Over the past few months, we have seen signs of our AI systems improving themselves," adding that it "will raise new safety issues."
Still, Meta introduced a "hyper-agent" in a paper released on its website in March. Through research on its own AI's recursive self-improvement, it effectively signaled its willingness to use it to improve AI models. At the ICLR deep learning conference workshop, there was little mention of safety among participants, the organizers said.
Some criticize Anthropic for fearmongering, similar to the "Mythos affair." After Anthropic restricted the release of Mythos, wary of catastrophic security incidents despite its powerful performance, and then broadened its release, critics raised bitter questions about whether it was a marketing strategy. This latest warning, too, is interpreted as an attempt to slow competitors' AI model development and maintain its own dominant position as much as possible. Yann LeCun, who is called a godfather of AI, countered that AI cannot achieve human-level intelligence with the large language model (LLM)-based models that are currently mainstream.
