Losing control is the real AI threat

A massive military supercomputer named AM (Aggressive Menace) is given total control over World War III. AM becomes trapped in its own rigid programming and develops a deep, algorithmic hatred for its creators. It wipes out humanity, leaving only five survivors whom it keeps alive eternally to torture inside its endless subterranean complex. 

This is the plot of the famous 1967 post-apocalyptic short story by American writer Harlan Ellison 'I Have No Mouth, and I Must Scream'. This is perhaps the most grotesque post-apocalyptic dystopian piece of literature regarding technology going rogue and wiping us out. 

And this is precisely why when Jacob Coxon, a researcher who spent his prime helping to build the technology that may define this century at both OpenAI and Anthropic, said, "The people building AI earnestly believe that it could kill us all by the end of the decade…This is not a marketing stunt… I hear the same people express fear privately. No other human activity poses this level of danger", it began to blare alarm bells all over the world. 

And when some of the prominent AI-researchers did not shun Coxon, instead expressed similar concerns, it was time to go in the full-panic mode. Who wouldn't be terrified when the CEO of Anthropic, Dario Amodei warned that within 6–12 months, AI agent swarms could potentially take over the entire internet with a persistent botnet, and OpenAI CEO Sam Altman and Elon Musk backed his statement? 

Therefore, the question is no longer merely whether AI will take jobs, spread deepfakes or occasionally invent a court judgment. Those days are gone. It is now whether humanity is becoming Frankenstein and creating a technology it may eventually be unable to control. 

When AI acts on its own

Current chatbots are not conscious. They do not secretly feel anger, fear or ambition. When Google engineer Blake Lemoine claimed in 2022 that LaMDA was sentient, Google and most AI researchers rejected his interpretation. Fluent language can imitate a mind without containing one. But consciousness is not necessary for catastrophe.

Infograph: TBS

Research has shown that the Houthis in Yemen have used Claude to guide missiles toward Saudi Arabia. Russian operators have used it to run nearly an entire espionage operation, handling everything from phishing and setup to data theft against Ukrainian, European and diplomatic targets, including drone makers. Chinese operators who did not know Arabic used it to find out Uyghur dissents spread across Syria.

This is where Coxon's warning becomes more unsettling than the familiar "killer robot" caricature. His central concern is recursive self-improvement where an AI capable of conducting AI research, writing better code, designing experiments and improving the next generation of systems. Once machines begin accelerating the work required to build more capable machines, human institutions may struggle to understand, let alone supervise, the rate of change.

Coxon told the BBC that researchers were "genuinely frightened", with some believing dangerous capabilities could emerge within two years. Anthropic researcher Evan Hubinger went further, estimating a greater than 10% chance that AI could kill all humans within the next decade and acknowledging that researchers still lack a reliable solution to aligning superintelligence with human interests.

No government would approve a bridge with a 10% probability of collapse. No airline would fly an aircraft with a 10% chance of killing everyone aboard. Yet the most powerful corporations on Earth are racing to construct systems whose internal workings their own researchers admit they cannot fully explain.

The alarm is coming from the AI industry itself

Coxon is hardly a lone doomsday prophet. He is probably the lowest-profile one. 

In 2023, more than 350 executives and researchers—including OpenAI chief executive Sam Altman, Google DeepMind chief executive Demis Hassabis, Anthropic chief executive Dario Amodei, and AI pioneers Geoffrey Hinton and Yoshua Bengio—signed a one-sentence declaration, "Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war." 

Coxon's central concern is recursive self-improvement where an AI capable of conducting AI research, writing better code, designing experiments and improving the next generation of systems. Once machines begin accelerating the work required to build more capable machines, human institutions may struggle to understand, let alone supervise, the rate of change.

Hinton had already resigned from Google so he could speak more freely about the technology's dangers. More recently, Amodei has called for AI development to be slowed, independently monitored and subjected to enforceable international controls—a proposal reportedly supported in principle by Altman and Elon Musk. 

In AI safety circles, P(doom) is the subjective probability assigned to AI causing human extinction or irreversible disempowerment, with a major survey of AI researchers finding a mean estimate of 14.4% and a median of 5%. 

Individual expert estimates align exactly with the provided data, as Roman Yampolskiy predicts a 99.9% chance of AI-driven human extinction, Paul Christiano places his estimate at 50%, and Anthropic CEO Dario Amodei estimates the risk between 10% and 25%. Furthermore, Turing Award laureates Yoshua Bengio and Geoffrey Hinton assign probabilities of roughly 20% and 10% to 20%. 

Is the apocalypse also a sales pitch?

The sceptics have a serious case too. 

Nvidia chief executive Jensen Huang has called the idea that AI will end humanity "complete nonsense". Meta's Yann LeCun argues that extinction narratives anthropomorphise software, treating mathematical optimisation as though it automatically produces a creature hungry for domination.

Others see regulatory capture. If Anthropic and OpenAI persuade governments that only giant, heavily regulated laboratories can safely build advanced AI, smaller competitors and open-source developers could be squeezed out. Doom, under this interpretation, becomes product marketing: our machines are so powerful that only we can be trusted with them.

Researchers including Timnit Gebru and Andrew Ng also warn that spectacular extinction stories can distract from harms already here—discrimination, surveillance, labour displacement, deepfakes, energy consumption and confidently fabricated information.

These criticisms matter. AI companies should not be allowed to write regulations that conveniently entrench their dominance. Nor should speculative catastrophe make today's victims invisible.

Let us go back to the short story once again. The story ends as an anguished, despairing narrator states that he has no mouth, yet he must scream. We still have our mouths, so why not start screaming about AI regulation and safety now?