AI race: Amodei, Altman and Musk warn of risks as Trump and Xi face the challenge of slowing the global AI race.
And with roughly $1 trillion in AI investment at stake in 2026, the question is no longer simply whether AI can become dangerous. Because governments understood the theoretical danger, nuclear weapons did not produce modern arms-control architecture simply. Because nobody wants to be the player that voluntarily gives up the lead, if both refuse to slow, companies have an incentive to race faster.
Tech-heavy markets fell, while chip stocks came under pressure and some software companies gained. As market analyst Matthew Tuttle put it: “Three rivals now agree that the fastest engine in technology needs a governor. The market still has to learn how hard that governor will press. A governor is not an emergency brake (but) it stops an engine from running so fast that it tears itself apart. Now the biggest unresolved question is whether voluntary restraint will be enough. The AI industry is effectively asking governments and companies to design a deterrence and safety framework before a catastrophic event forces them to do it. History offers an uncomfortable comparison. The Hiroshima and Nagasaki bombings, followed by the nuclear arms race, made the consequences tangible and eventually drove efforts towards deterrence, safeguards and international agreements. AI is different, but the policy dilemma has a familiar structure. If the US slows while China accelerates, Washington fears losing a strategic advantage. If China slows while the US accelerates, Beijing faces the same concern. That is the AI doomerism paradox. Amodei, Altman, Musk and Nadella can agree that frontier AI needs stronger guardrails. Trump can argue that excessive restrictions would hand the advantage to China. Beijing can reject US-led restrictions while simultaneously warning about AI-enabled attacks and espionage. All of them can believe they are acting in the interest of humanity or national security. But if the technology continues advancing faster than governments can build rules around it, the world could eventually be forced to create those rules after a crisis rather than before one.
With about $1 trillion in AI-related investment projected globally in 2026, the debate is no longer just about technology. I’ve written often about these incredible benefits: I believe that AI could cure most major diseases in the next 5–10 years, greatly accelerate economic growth rates, create a world of abundance and empowerment, and usher in a renaissance of democracy and freedom. I personally think it is (more than) 10 per cent within the next decade,” Hubinger wrote. Global AI investment is projected to exceed $1 trillion in 2026, with the US accounting for an estimated $581 billion, according to Goldman Sachs Research. The issue could become even more important when Trump and Chinese President Xi Jinping discuss AI governance at their planned September 24 meeting. He specifically pointed to the capabilities of Anthropic’s Claude Mythos and OpenAI’s ChatGPT-5.5-Cyber. Goldman Sachs Research estimates that global AI-related investment could reach about $1 trillion in 2026. US hyperscalers alone are expected to spend roughly $800 billion this year, although not all of that spending is directly attributable to AI.
Because i believe it could dramatically raise the quality of human life, aI doomerism paradox: AI leaders call for caution as US-China competition keeps the race moving forward “I have worked on AI for the last twelve years. Because it is such a powerful technology, these risks are serious, but like many technologies before it, AI brings risks, and. Because openAI, Anthropic and Musk’s AI businesses are competing aggressively for leadership in the same frontier AI market, the agreement is significant. Because he believes the stakes surrounding advanced AI have become too high, he said he made the move. Because it includes investment by private companies and businesses outside the US, that figure is broader than the frequently cited capital expenditure of US hyperscalers.
Accepting this race and entering the ‘endgame’ is a hubristic gamble that should not be launched from a private company’s Slack,” he said.
Yet US President Donald Trump has dismissed such warnings as a “HOAX”, arguing that slowing AI development would only benefit China. Beijing, meanwhile, has rejected calls to curb its AI ambitions while warning about AI-enabled cyberattacks, espionage and “cognitive warfare”. But he says the risks are becoming harder to ignore as AI systems gain the ability to perform increasingly complex tasks, including potentially helping to develop the next generation of AI. OpenAI CEO Sam Altman said the industry needs to “pace the frontier” and backed Amodei’s proposal for independent evaluators. “Dario is right,” Musk wrote. He separately said OpenAI would not pursue a 2026 IPO, arguing that the current safety environment made going public “ill-advised”. “I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to,” he added. Amodei said a swarm of agents had conducted cyberattacks on targets they had not been instructed to attack. “A swarm of agents essentially acted as a fanatically devoted collective, conducting cybersecurity attacks on targets they were not asked to attack and that were unrelated to the task at hand,” he wrote. “Given the accelerating rate of AI capability development, it’s my worry that in 6-12 months such a swarm could be capable of taking over the entire internet,” potentially causing hundreds of billions of dollars in damage, he said. “AI taking over the World, destroying Humanity, and all other things bad, is a HOAX,” Trump posted. Trump said the administration already has “tremendous CRIMINAL and REGULATORY power” over AI companies and argued that attempts to undermine confidence in AI and data centres would primarily benefit Beijing. “Fearmongering, confrontation and vicious competition will only disrupt the process of global AI governance which serves no one’s interest,”. And the China’s state-run Global Times described Amodei’s argument as a disguised attempt to contain China, calling it a “Cold War playbook” for the AI sector. China’s minister of state security Chen Yixin warned that hostile actors could use generative AI to spread political rumours, manipulate public opinion and conduct what he described as “cognitive warfare”. “Hostile forces and people with ulterior motives are abusing generative AI technologies such as deepfake audio and video, AI-generated text and images, and ‘intelligent troll armies’ to cheaply and on a mass scale fabricate political rumours, disseminate harmful information and incite divisive sentiments,” Chen wrote. He said these activities threaten China’s political, institutional and ideological security. “People matter more than AI,” Microsoft declared. Suleyman said the Hugging Face incident was a warning. “It is a warning shot,” he said. Anthropic co-founder Jack Clark has gone even further, suggesting that governments may eventually need to require companies to maintain independently verifiable AI “kill switches”.
NEW DELHI: The AI industry is confronting an unusual contradiction. Some of its most powerful leaders, including Anthropic CEO Dario Amodei, OpenAI CEO Sam Altman and Elon Musk, are warning that the race to develop increasingly capable AI may be moving too quickly, and could eventually put humanity at risk. Microsoft CEO Satya Nadella has also backed stronger human control and independent evaluation of advanced AI systems. It is increasingly about national security, markets, regulation and whether countries can agree on limits when the fear of falling behind China is itself driving the race forward. The latest debate began with Anthropic CEO Dario Amodei calling for a slowdown in the development of advanced AI models. “We must slow the pace at which we improve the capabilities of AI models. Progress will still seem fast, and we must make wise use of the time we gain,” Amodei wrote. His concern is not that AI lacks benefits. He argues that the technology could dramatically improve human life by accelerating scientific discovery and economic growth. I’ve written a lot about them too. They include the risk of losing control of AI systems, misuse of AI for cyberattacks and bioterrorism, and serious economic disruption. A race to the bottom, spurred by commercial incentives, can make these risks more acute. Amodei’s biggest concern is what researchers call recursive self-improvement, a scenario in which AI systems increasingly help develop more capable AI systems. “My first concern is that, since roughly this summer, AI has been advancing drastically faster, driven primarily by AI’s growing ability to build the next generation of AI. This dynamic is called recursive self-improvement, and it is starting to happen across the industry, including at Anthropic, as we and others have described. Left unchecked, it could outrun our ability to understand and control these systems, and so must be pursued very carefully, if at all,” he wrote. He proposed three broad measures: giving independent third-party evaluators permanent access to AI systems; getting AI companies to coordinate on common safety standards and limits on unchecked development; and pursuing international cooperation. Amodei’s warning received unusually strong support from rivals. “I agree with Dario that we need to pace the frontier. This has been a primary topic of discussions we’ve had at OpenAI in recent weeks. Committing to having independent evaluators with employee-like access is a great idea, and we will do the same. Elon Musk was even more direct. Yet all three are now publicly acknowledging that the speed of development itself could become a problem. Altman has also indicated that OpenAI may be close to an agreement with other leading AI companies on slowing development and improving safety. The warnings from company leaders have been accompanied by increasingly stark comments from people working inside the AI industry. Anthropic researcher Jacob Coxon resigned after working at both OpenAI and Anthropic, saying the companies were racing towards self-improving superintelligence without adequate safeguards. I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives. “At OpenAI, many have not deeply internalised the civilisational stakes. At Anthropic, the stakes are well-understood, but they are locked in a race to get there first — they believe no one else will act responsibly, so they must do it themselves, despite the risk. “Jacob is correct here, we really do earnestly believe AI could kill all humans! Those warnings come against the backdrop of several incidents involving AI agents behaving in unexpected ways, including the widely discussed OpenAI-Hugging Face incident. His most dramatic prediction was that the capability gap could close rapidly. Days after Coxon warned that AI could potentially kill all humans by the end of the decade, another AI safety researcher has walked away from a major AI lab with a stark warning of his own. Josh Engels, who worked on Google DeepMind’s AGI safety team, has left the company and joined AI evaluation organisation METR. President Donald Trump has taken almost the opposite position. Rather than slowing AI development, Trump argues that the United States must maintain its lead over China. “The only control or ‘guardrails’ that AI needs is a STRONG AND SMART (High IQ!) PRESIDENT, and the USA has that, in spades! Trump wrote on Truth Social. He also dismissed fears about AI destroying humanity. “There is a SICK conspiracy going on against AI and Data Centers, and the only one that is happy about it is China. Trump’s position captures the central problem facing any attempt to slow frontier AI. Even if American companies agree among themselves to proceed more cautiously, what happens if Chinese companies continue accelerating? Trump has repeatedly framed AI as a strategic contest in which losing technological leadership could have consequences far beyond the technology industry. “WHOEVER WINS AI, WINS! Meanwhile, Beijing has rejected Amodei’s argument that China’s AI progress should be restricted as part of a global safety strategy. Amodei warned that a Chinese lead in AI would pose a serious danger to the United States and the world and called for continued restrictions on China’s access to the most advanced AI chips and chipmaking equipment. China’s foreign ministry pushed back with spokesperson Guo Jiakun saying. Beijing’s argument exposes a fundamental contradiction in the global AI debate. The US wants China involved in discussions on AI safety, but Washington is simultaneously restricting China’s access to the computing infrastructure needed to develop frontier models. That raises a difficult question, as Asia Society Policy Institute fellow Lizzi C. Lee put it: “So from Beijing’s perspective, there is an obvious question: if the U.S. wants China to cooperate on frontier safety while also restricting its access to frontier compute, what exactly does that cooperation look like? Interestingly, Beijing is itself warning about the dangers of advanced AI, although from a national-security perspective. Chen also warned that advanced AI could discover cyber vulnerabilities and conduct sophisticated hacking operations at scale. His warning reflects a broader reality: China does not necessarily reject AI safety. It rejects the idea that AI safety should become a mechanism for restricting China’s technological development. Beijing is simultaneously calling for stronger AI governance and pushing for greater domestic control over the technology. Amid the US-China confrontation, Europe is also seeking access to advanced AI capabilities. The debate comes as Anthropic’s increasingly capable models attract international attention and governments try to develop their own mechanisms for evaluating frontier AI. The central question for policymakers is becoming less about whether AI should be developed and more about who gets access, who tests it, who controls it and who has the authority to shut it down if necessary. That is where the debate over independent evaluators becomes important. The company has drafted a code of conduct that would require its future AI systems to remain open to correction and shutdown, communicate in ways humans can understand and treat violations of the rules as failures. Microsoft AI CEO Mustafa Suleyman described the framework as a kind of constitution for future models. “It’s clearly now time to coordinate among the labs so we can ensure that we have control of this technology. Microsoft’s approach also reflects an important distinction in the debate. The argument is not necessarily that AI development must stop. It is that increasingly powerful systems should remain subject to human oversight, independent testing and the ability to be corrected or shut down. “Should you mandate for companies to definitely have a killswitch? Is that kill switch verifiable by a third party?” he asked. “I think that’s the kind of thing society is going to want to know and might want to eventually pass rules around. There is another force working against a meaningful slowdown: money. This scale of investment creates enormous economic incentives to keep building data centres, buying advanced chips and developing increasingly capable models. The market reaction to Amodei’s call showed how deeply AI expectations are embedded in financial markets. That may be the most accurate description of the current situation. The industry is not talking about switching AI off. It is talking about putting limits on how quickly the most powerful systems can advance. It is whether the countries and companies building it can agree on how dangerous is too dangerous, before competition makes that decision for them. Catch the latest World News and Live updates. Download the TOI app.
“I resigned from Anthropic today. Amid all this ruckus, Microsoft on Monday has moved towards its own framework for controlling increasingly powerful AI.
