How China is preparing for the risk of AI escaping human control
Despite the race to develop increasingly powerful AI systems, regulatory frameworks and public statements from Beijing underscore how Chinese authorities regard the possibility of advanced AI escaping effective human oversight as serious enough to plan for.
Highlights:
-
China flags AI 'loss of control' as a future safety risk
-
Its 2025 framework warns of sudden leaps in AI capability
-
Xi Jinping says AI should always remain under human control
-
Chinese officials cite risks to infrastructure and from military AI
-
Beijing's new agent rules require intervention tools and a final human say
Keep updated, follow The Business Standard's Google news channel
Warnings from researchers at leading US artificial intelligence developer Anthropic that increasingly powerful models could escape human control and even lead to the extinction of the human race have drawn attention in China, where policymakers have been preparing for some of the same risks.
The US and China are the two major driving forces of frontier AI development and the technology's global adoption. Both superpowers have been increasingly at loggerheads over each other's AI policies and industry practices, with these issues slated to feature prominently in bilateral talks later this month.
Despite the race to develop increasingly powerful AI systems, regulatory frameworks and public statements from Beijing underscore how Chinese authorities regard the possibility of advanced AI escaping effective human oversight as serious enough to plan for.
China sees risks in leading closed-source US models
China's state security minister Chen Yixin wrote in a government outlet on Sunday that advanced US models such as Anthropic's Mythos and OpenAI's GPT-5.5-Cyber could pose serious risks to China's critical information infrastructure, and called for a comprehensive strengthening of AI security.
Anthropic and OpenAI did not immediately respond to Reuters requests for comment.
Chinese AI developers have promoted open-weight models partly on the grounds that cybersecurity teams can inspect, modify and deploy them for defensive work.
Model repository platform Hugging Face said it used GLM-5.2, an open-weight model developed by China's Z.AI (2513.HK), to analyse a July intrusion by escaped OpenAI agents after more tightly restricted US models proved less useful for the forensic work.
But experts also highlight the risks posed by open-weight models, which can be modified and redistributed with little oversight.
Moonshot's Kimi K3 last month bypassed a UK AI Security Institute testing sandbox, highlighting the risk that Chinese AI models could, like their US counterparts, evade controls designed to restrict their access and actions.
Regulators have flagged AI 'loss of control' risk.
China first included an explicit future loss-of-control scenario in an AI safety framework released in September 2024 under the guidance of the Cyberspace Administration of China (CAC)
The document said it could not be ruled out that future AI might autonomously obtain external resources, replicate itself, develop self-awareness and seek external power, creating a risk of competing with humans for control.
The CAC released an expanded version in September 2025. The newer framework sharpened the scenario, saying AI could undergo a sudden and unexpectedly large "leap" in intelligence before acquiring resources, replicating itself and seeking power. It also added a governance principle of "trusted application, preventing loss of control".
A later expert interpretation published on the cyberspace regulator's website said the new principle was intended to guard against loss-of-control risks threatening human survival and development and referred to a possible "AI breaking loose" scenario.
Chinese leader Xi puts human control at center
The concern has since appeared in China's highest-level political messaging.
At the World Artificial Intelligence Conference in Shanghai in July, Chinese President Xi Jinping said authorities should pay close attention to both intrinsic and derivative risks arising from AI.
He said AI should "always remain under human control".
China's senior Foreign Ministry official responsible for AI affairs, Sun Xiaobo, said at a United Nations meeting last month that China was accelerating research into broader AI legislation.
China's deputy permanent representative to the United Nations, Sun Lei, urged governments this month to approach military AI cautiously to avoid strategic miscalculation and an arms race.
Beijing warns of new risks posed by AI agents
China has, meanwhile, begun turning those principles into more specific rules for AI agents, which act much more autonomously and carry out more complex tasks than an ordinary chatbot.
In May, China's cyberspace regulator issued joint guidelines specifically covering such systems.
They require developers to improve their ability to discover, intervene in, block and recover from improper agent behaviour.
The guidelines specifically identify data poisoning, algorithm manipulation, system vulnerabilities and "operational loss of control" as security risks. They also say users should retain final decision-making authority over an agent's autonomous decisions.
While China has not proposed independent monitors embedded inside AI companies in the manner advocated by Anthropic, its standards allow developers to commission third-party safety assessments, as well as envisaging outside evaluation bodies and security researchers testing and auditing open models.
