A still from Terminator 2: Judgement Day
After the success of Endgame in 2019, Disney executives were thoroughly convinced that no matter what they produced, it would be lapped up by audiences if you stuck a Marvel logo on it. The cases predated Coxon’s resignation, but the timing was almost offensively perfect, rather like Nvidia agreeing to acquire Hugging Face for $12.93 billion days after OpenAI publicly disclosed the July incident. Because if we don’t, China will build it first, and I don’t want to get murdered by a computer that speaks Chinese, ray Kump, an American comedian, summed it up best: “We have to build AI that murders us,. Because the Chinese might get there first, in this case, the truth is that some of the people closest to building the world’s most powerful AI systems are now openly saying that those systems might kill all of us, while simultaneously explaining why they cannot stop building them. Because the proposed solution is regulation, the financial overlap is curious.
Bernie Sanders and Democratic congressman Greg Casar had already announced forthcoming legislation aimed at banning artificial superintelligence and pausing some advanced AI development. Anthropic released a threat report describing attempts to use Claude for cyber operations, weapons work and potentially dangerous biological research, including gain-of-function work involving chikungunya.
He thinks the existential threat is “greatly exaggerated”, but his point about Coxon was simpler: forget the biography and ask whether the man is actually right. Marvel Studios is as good a metaphor for what happens when a company starts believing in its own infallibility. Reality begged to differ, and audiences showed little interest in the post-Downey Jr woke universe till Kevin Feige used Jonathan Majors’ troubling behaviour to kill Kang and instead bring back the OG Godfather of the MCU: Robert Downey Jr and many of the original cast. And it’s fitting that the movie will be called Avengers: Doomsday , where Robert Downey Jr plays one of Marvel’s iconic villains. The jury — Reddit fanboys of Marvel — are trying to figure out how the geniuses at Marvel will tie it all in and whether Victor von Doom will be a variation of Tony Stark from a different universe. Speaking of geniuses and doomsday and companies that believe in their own infallibility, the part of the world that cares about AI beyond creating images that make us look like extras in Mithun Chakraborty movies has been wringing its hands over several questions: Is AGI lready here? By which year will it kill us all? What if the Chinese manage to create the ‘killer’ AI before Silicon Valley tech bros? That would be ridiculous. There’s no smoke without fire and, more importantly, there are very few good jokes without a sliver of truth. AI, AI, AI, AI Since generative AI entered our lives with the advent of the first iteration of ChatGPT, the word AI has been repeated ad nauseam . Capitalists have dreamt of companies where machines that don’t need smoke breaks can do all the jobs and only the CEO needs to be paid. Product managers have umpteen agents vibe-coding at the same time, building products no one asked for. The more cynical ones have pointed out that the point of AI was to do our laundry and boring chores, not write our poetry or essays. And there have been the doomers who have seen the advent of AI as the next rung in evolution, where humans are no longer required and the human population would decline the way horses did after the advent of cars. And then there are the pro-max doomers who believe the prophecy made in every Hollywood movie where AI rises up to enslave all humans till an Austrian actor with one catchphrase travels back in time to stop it. Add the political backdrop and the timing looks suspiciously neat. If frontier AI eventually requires enormously expensive safety testing, auditing, cybersecurity and government compliance, OpenAI, Anthropic and Google can pay for it. Two blokes with a promising model and a WeWork membership probably cannot. A safety barrier can also become a moat. The Rosa Parks Paradox So, there is a network, remarkable timing and legitimate questions, but not proof of a conspiracy. The strategic choice of messenger did not make segregation any less real. Gary Marcus, one of OpenAI’s greatest critics, makes the most logical yet sceptical argument. And just when the whole thing starts looking too perfectly packaged, reality inconveniently gives the doomers something to work with.
Rosa Parks was not an accidental symbol either: her refusal to surrender her seat was not staged, but civil-rights organisers had been looking for a strong test case against bus segregation and deliberately rallied around Parks after earlier cases, including Claudette Colvin.
Because incentives matter, perhaps the question here isn’t existential but what the Romans called cui bono , or who benefits,. Because how else will we know how to defend ourselves, hardly makes the answer easier, the behaviour of AI companies, from OpenAI, which started as a non-profit, to whatever it is today, to Anthropic, which built itself around AI safety while also being the guy building the killer robot. Because the system supplies the justification, nobody inside it needs to be lying. Because fear brings money, attention and regulation, perhaps Big Tech is hyping doomsday. Because after all the multiverse nonsense and speculation about how Robert Downey Jr is going to return to the MCU as Doctor Doom, one really wants to watch Kevin Feige explain that before artificial intelligence kills everyone, hopefully, whatever version of doomsday the AI world is worried about waits until after Avengers: Doomsday ,.
Anthropic also revealed that a weapons-development cell in northern Yemen had used Claude Code to help develop guidance, navigation and control software for a guided rocket, a long-range ballistic missile and a hypersonic-glide variant.
Max Weber, the German sociologist who studied capitalism and bureaucracy, used the idea of the “iron cage” to describe how rational institutions can eventually trap the people inside them. The cell even test-fired the guided rocket, apparently failed, and returned to Claude to figure out what had gone wrong. Cui Bono? Someone can profit from pointing out that your house is on fire, even while the fact remains that perhaps your house is on fire. American author Upton Sinclair famously observed that it is difficult to make a man understand something when his salary depends upon his not understanding it. AI has produced the funhouse-mirror version: what happens when your salary, your company valuation and your sincere belief that you are saving humanity all depend upon understanding the world in exactly the same way? The optimists say AI will cure diseases, create abundance and even solve Navier-Stokes, though how that improves our lives is beyond me. The doomers say the same technology might destroy civilisation. Opposite advertisements, same product: AI is the most important thing in the world. One tells investors they cannot afford to miss it; the other tells governments they cannot afford to ignore it. And the jokers point out that the idea of being killed by a Chinese robot is ridiculous. The world-ending — as all Hollywood movies have predicted — should be conceived in America. Anthropic has reached a similar bind. It built itself around AI safety but believes meaningful safety research requires remaining near the frontier. Everyone might be following perfectly logical rules from his own position, but the system itself begins deciding what behaviour is rational, making it increasingly difficult for anyone inside it to simply stop and we appear to be in the iron cage now. Perhaps they believe humanity is playing Russian roulette. Or perhaps all of them are partly right or wrong. Human beings have always been good at discovering that whatever advances our interests also happens to be morally necessary. As the saying goes, once you have got them by the profit sheets, the hearts and minds will follow. The irony of AI alignment may be that we spent years worrying about machines rationalising their objectives while the species building them has been practising that trick since we came down from the trees. And after that, if one has to be killed by a robot, let it be an American one (with an Austrian accent), not a Chinese one. What could be more ridiculous than that? Get the latest technology news and updates. Download the TOI App.
Evan Hubinger, an Anthropic researcher who leads work on AI alignment, then gave the headline that all journalists seek: AI’s chance of killing humans within a decade, according to him, was above 10%. Say someone says p(doom) is 10%. Applied to AI, imagine someone tells you there is only a 1% chance that pressing a button will kill every human being on Earth. You may have no idea whether that 1% estimate is accurate, but the consequence is so catastrophic that pressing the button anyway would still be an extraordinary gamble. And like all products that were conceived in a start-up garage before convincing people to part with $3,000 to look cool, doomsday apparently comes with a supply chain. As Parker Thayer, an investigative researcher at the conservative Capital Research Center, pointed out, The Wall Street Journal published an exclusive with Coxon 18 minutes before his X thread appeared. The Survival and Flourishing Fund’s own records list Jaan Tallinn, the Skype co-founder and long-time existential-risk philanthropist, as the funding source behind more than $2 million for the AI Futures Project in 2025, $516,000 for Encode AI and $1.635 million for the AI Policy Institute. Tallinn also led Anthropic’s original $124 million funding round in 2021.
One report in Axios buttressed the seriousness of Coxon’s claims by pointing out that he left Anthropic two months before his equity was due to vest, a sure sign of a sudden awakening of conscience before payday. Because humanity has not made a superintelligence before, not unlike the famous Doomsday Clock for nuclear winter, which was also never intended to represent a mathematically calculated probability of extinction, there is no historical sample to calculate it from. The philosophical objection to this is Pascal’s wager, named after French mathematician and philosopher Blaise Pascal, who argued that it didn’t matter if God existed or not: the safer option was to assume the existence of a higher deity simply because, why not?
The Latest Doomsday The latest doomsday alarm was raised by Jacob Coxon, a 27-year-old AI researcher who worked at OpenAI and Anthropic before resigning from the latter this week, arguing that frontier labs are “gambling with our lives” by racing towards self-improving AI, where machines keep building more capable successors until humans become like journalists in maths class and can no longer understand or control the process. Samuel Marks, another Anthropic researcher, said that labs keep building because of commercial incentives and the fear that “less responsible competitors” will get there first, which for OpenAI is Anthropic, and for everyone in Silicon Valley is China. Geoffrey Irving, who worked at DeepMind, OpenAI and Google Brain, puts the chance of everyone dying because of superintelligence at roughly 50% over the next few to ten years, while admitting that he does not expect to know whether that number should really be below 10% or above 90% until “we either make it through, or we don’t”. Professor Geoffrey Hinton, the ‘Godfather of AI’, who won a Nobel Prize in Physics (for AI work, for some reason), said that a 10% chance of AI killing all humans within a decade was “not an unreasonable estimate”, leading a BBC anchor to lose her stiff upper lip and say: “Oh my God.
Others have wondered if the doomers weren’t hyping up their own products a little too much, given that the entire AI industry is rather precariously placed from a financial perspective and could come crashing down like the proverbial Tower of Babel if companies don’t keep adding bricks of money to its edifice. Which brings us to the question so obvious that even the most basic chatbot would ask it: if you genuinely believe that a machine you are building could kill humanity, why keep building it? The Manhattan Project Logic About a century ago, that logic was used for the Manhattan Project: if Nazi Germany got the bomb first, the Allies argued, it would be catastrophic for the world. The Japanese would probably disagree. Game theory calls this the prisoner’s dilemma: everyone would be safer if everyone slowed down, but no one wants to let someone else get there first. And unlike the Manhattan Project, this time the physicists have stock options. Image: Philippe Halsman/Magnum Photos Now there are obvious sceptics, coming from a culture that has existed for millennia and sometimes wonders if we are only Brahma’s dreams. Princeton computer scientist Arvind Narayanan and AI researcher Sayash Kapoor scoff at the concept of p(doom), shorthand for the probability that AI will free us of our burdens and wipe us out. it was imagined like any AOP figure While the number sounds as though it has been calculated from data. The numbers hardly make that easier. Which might be the most honest description of p(doom) yet. Pascal’s wager or Pascal’s mugging? That is essentially the disagreement: Narayanan and Kapoor question where the number comes from, while the doomers argue that when the possible outcome is extinction, even an uncertain number cannot simply be ignored. Pascal’s wager has a less famous cousin called Pascal’s mugging, which asks what happens when tiny probabilities and enormous consequences become too easy to exploit. If someone tells you there is a one-in-a-trillion chance he will destroy a trillion planets unless you pay him, mathematics can make paying look rational even while common sense screams that you are being conned. Which brings us back to the other half of the question: what if Big Tech, or at least the ecosystem around AI safety, is selling us doomsday? There are some things about the chain of events that would make everyone wonder what’s going on. The people who amplified Coxon’s thread were Nathan Calvin of Encode AI, Peter Wildeford of the AI Policy Network and Daniel Kokotajlo of the AI Futures Project. Then there is the money trail. The AI Policy Institute is separate from the AI Policy Network, but both were founded and are run by Daniel Colson.
