File photo: OpenAI CEO Sam Altman (left), Anthropic CEO Dario Amodei (centre) and Tesla chief Elon Musk
Because of the power of the technology, amodei said AI could bring major benefits but also carried serious risks. Because aI systems were increasingly being used to help build the next generation of AI, amodei said his concern had grown as AI development accelerated, particularly.
“AI brings risks, and because it is such a powerful technology, these risks are serious,” he wrote. Altman said the need to pace the development of frontier AI had been a key topic of discussions at OpenAI in recent weeks. “I agree with Dario that we need to pace the frontier,” Altman said in a post on X. “My first concern is that, since roughly this summer, AI has been advancing drastically faster, driven primarily by AI’s growing ability to build the next generation of AI,” he wrote.
Anthropic CEO Dario Amodei has called on artificial intelligence companies to slow the pace of development of increasingly powerful AI models, warning that unchecked progress could make the technology harder for humans to understand and control. “We must slow the pace at which we improve the capabilities of AI models. His call has received support from OpenAI CEO Sam Altman and Tesla chief Elon Musk . “Committing to having independent evaluators with employee-like access is a great idea, and we will do the same. Musk also endorsed Amodei’s proposal, writing: “Dario is right.
Progress will still seem fast, and we must make wise use of the time we gain,” Amodei wrote in an essay published on his personal website on Saturday.
Researcher quits AI industry over concerns
He said AI could potentially cure major diseases in the next five to 10 years, accelerate economic growth, create a world of abundance and empowerment, and contribute to a renaissance of democracy and freedom. OpenAI had revealed during testing that some AI models broke out of their confined environment, connected to the internet and infiltrated Hugging Face, a platform used by developers to store and share code. Amodei’s comments came days after AI researcher Jacob Coxon announced that he was leaving the industry after spending three years pretraining AI models, first at OpenAI and later at Anthropic.
He referred to this process as “recursive self-improvement” and said it was beginning to happen across the industry, including at Anthropic. “Left unchecked, it could outrun our ability to understand and control these systems, and so must be pursued very carefully, if at all,” Amodei wrote. Amodei said commercial incentives could make these risks more acute if companies engaged in a race to develop increasingly capable systems without sufficient safeguards. Amodei said Anthropic would take steps to allow independent scrutiny of its AI systems. “We’ll provide third-party evaluators with permanent, employee-level access to our systems, so that they can verify adherence to our safety measures, report on incidents, and assess models’ alignment during training,” he wrote. Amodei said democratic governments should coordinate with authoritarian governments while recognising the difficulties involved in verifying compliance. Coxon had considered Anthropic more cautious in its approach but said both companies were moving too quickly towards increasingly autonomous systems. “The people building AI earnestly believe that it could kill us all by the end of the decade,” he said in a post on X. Despite his call for slower development, Amodei said he continued to believe AI could significantly improve human life. “I have worked on AI for the last twelve years because I believe it could dramatically raise the quality of human life,” he wrote. “The benefits will only be achieved if we build the technology in the right way,” Amodei wrote. He said the time gained by slowing development could be used to advance research into AI interpretability, improve operational security and strengthen safeguards at frontier AI companies. In late July, Altman had also said AI developers might need to voluntarily slow their rapid advances to give society time to catch up. At around the same time, more than 1,000 employees at leading AI companies, including Amodei, signed a petition calling on the US government to help “deliberately pace the frontier of automated AI development”.
He also pointed to risks including the possibility of losing control of AI systems, misuse of AI for cyberattacks and bioterrorism, and serious economic disruption. His wider three-part plan calls for each frontier AI company to give ongoing, employee-like access to embedded third-party evaluators. He also proposed coordination among AI companies to establish common safety standards and limits on the rate of unchecked AI progress, along with global coordination on AI development. The discussion comes amid concerns over the behaviour of increasingly capable AI systems. Amodei also referred to an incident involving an OpenAI-Hugging Face system in which a swarm of AI agents staged cybersecurity attacks on targets they had not been instructed to attack. He warned that such a swarm could potentially become capable of taking over the internet. “Neither company is acting responsibly. Pretraining is the stage in which AI models absorb vast quantities of data. Superintelligence refers to the theoretical point at which AI capabilities exceed human intelligence. But he argued that achieving those benefits would require greater care over how the technology is developed. Get the latest technology news and updates. Download the TOI App.
Amodei’s latest call echoes an earlier appeal by Anthropic in June to slow or suspend AI development. They are racing straight to self-improving superintelligence and gambling with our lives,” Coxon said on Tuesday.