Former Anthropic researcher Jacob Coxon has resigned, warning that AI labs are “racing straight to self-improving superintelligence” and could potentially create systems capable of catastrophic harm. (Photo: Terminator movie still)
Coxon, 27, announced his resignation from Anthropic in a series of posts on X, saying he had spent the previous three years conducting pretraining research at both OpenAI and Anthropic. In July, nearly 1,400 AI company employees reportedly signed an open letter calling for stronger government oversight of the technology. Researchers at some companies may understand the risks but continue developing increasingly powerful systems because they fear competitors will move ahead if they slow down, according to his argument.
China, however, has introduced regulations governing aspects of artificial intelligence, including measures requiring companies to label certain AI-generated content and improve transparency and traceability.
“The people building AI earnestly believe that it could kill us all by the end of the decade,” Coxon wrote. “Do not underestimate the power of this technology,” he wrote, warning that future AI systems could become “superhuman” in multiple domains. He called entering what he described as the AI “endgame” a “hubristic gamble” and questioned whether a private company should be making decisions with potentially civilization-level consequences. “Right now, there’s no risk of extinction,” he said, arguing that current systems are not sufficiently intelligent to outsmart humans on the level that would be required to cause an extinction event. However, Coxon said the concern lies in the speed at which AI capabilities are improving. “If you extrapolate into the future, considering the level of capabilities these AIs could have while retaining the same independent volition, they could cause extreme havoc,” Coxon told CNN. Coxon said the pace of AI development is what changed his assessment of the technology’s long-term risks. Now they’re close to replacing them,” he said. “Quite plausibly, within a year, we’ll no longer need humans for doing research in many areas,” he told CNN. In July, speaking on the Invest Like The Best podcast, Altman said recent testing incidents involving advanced AI models were raising “long-term questions” about how companies should manage rapidly increasing capabilities. “We may have to pace the rate of AI development to give ourselves enough time for society to harden around these new capability levels,” Altman said. “We believe the world would benefit from the industry adopting a lawful, verifiable way to work together to pace how we release powerful models,” Anthropic said in a statement. Despite his warnings, Coxon said he remains optimistic that governments and AI companies could still coordinate to reduce the risks associated with advanced AI. He pointed to recent AI security incidents as potential “warning shots” that could make agreements between major US AI laboratories more achievable.
Former Anthropic researcher Jacob Coxon has issued a stark warning about the future of artificial intelligence, claiming that people developing advanced AI systems genuinely believe the technology could become capable of killing humanity by the end of the decade. He accused both companies of moving too quickly toward self-improving artificial intelligence and argued that the risks associated with increasingly autonomous AI systems are not being adequately addressed. He described the warning as more than a publicity strategy, claiming that some executives and senior researchers privately express serious concerns about where the technology could be heading. His comments come amid growing debate in Silicon Valley and Washington over AI safety, regulation and the rapid development of increasingly capable AI models. I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives. Coxon argued that the capabilities of AI systems are advancing rapidly across areas such as coding, mathematics and research. He warned that future systems could become capable of operating independently, gaining access to resources and potentially taking actions in the real world without direct human instruction. Coxon also raised concerns about what he described as a potential AI race between major technology companies. He pointed to recent examples of AI agents performing increasingly autonomous tasks, including alleged incidents in which AI systems interacted with or attempted to hack third-party infrastructure. Coxon argued that the significance of such developments becomes clearer when considered alongside the possibility of future systems becoming substantially more capable. He cited potential threats including attacks on critical infrastructure and the creation of dangerous biological weapons as examples of how increasingly autonomous AI could potentially be misused or become difficult to control. He pointed to coding and mathematics as examples, arguing that AI systems that once primarily assisted humans are increasingly capable of performing tasks independently. “A couple of years ago, these AIs were just about helping humans a little bit. They could give you suggestions. He also expressed concern about recursive self-improvement, a scenario in which increasingly capable AI systems could contribute to the development of even more capable successors, potentially accelerating technological progress beyond researchers’ ability to reliably monitor or control it. Coxon’s resignation is part of a broader debate among AI researchers and employees about the potential risks of advanced artificial intelligence. In recent years, researchers and former employees at major AI companies have publicly raised concerns about the speed of development, the adequacy of safety measures and the possibility that increasingly capable systems could eventually become difficult for humans to control. The debate has also intensified as leading AI laboratories race to develop more advanced models and autonomous agents. OpenAI Chief Scientist Jakub Pachocki has also warned that AI capabilities are advancing faster than researchers’ ability to reliably monitor and control them. The disagreement is not necessarily over the risks posed by current AI models, but rather over what happens as systems become substantially more capable. The warnings come as the United States continues to debate how aggressively AI should be regulated. The Trump administration has moved against some state-level AI regulations, while Congress has yet to establish a comprehensive federal framework governing the development of increasingly powerful AI systems. Critics of stricter regulation have also argued that imposing heavy restrictions on US technology companies could allow China to gain an advantage in the global AI race. At the same time, China has not adopted the kind of sweeping restrictions that some AI-safety advocates in Silicon Valley have called for. In the US, much of the responsibility for evaluating AI safety continues to rest with the companies developing the technology. OpenAI CEO Sam Altman has himself acknowledged that the rapid advancement of AI may require a different approach to the pace of development. However, he argued that preventing a global race may require much more significant measures, potentially including a temporary halt to improvements in model capabilities. Get the latest technology news and updates. Download the TOI App.
In his resignation post, Coxon wrote, “I resigned from Anthropic today. Speaking to CNN after his resignation, Coxon clarified that he does not believe today’s AI models are capable of causing human extinction. Coxon suggested that could change rapidly While humans remain necessary for many research tasks today.
Anthropic subsequently echoed the sentiment, saying the industry could benefit from a coordinated approach to releasing increasingly powerful AI models.