WeChat, a messaging app, is practically part of the national infrastructure in China, woven into daily communications, government services and digital payments.
That’s why a recent discovery by a small team of researchers in California — that a tool built with artificial intelligence could breach millions of accounts in just a few hours — has rattled experts and Chinese analysts. It proved that with A.I., even a tiny team with no government backing can mount a devastating attack on an app used by 1.4 billion people a month.
“From a destruction standpoint, it’s extremely potent,” said Zhao Minghao, deputy director of the Center for American Studies at Fudan University in Shanghai. He said the ability to take down an app like WeChat, given its importance to the Chinese public and its huge user base, amounted to “a new kind of nuclear weapon.”
This discovery adds fuel to warnings from researchers that A.I. hacking capabilities are advancing faster than defenses can keep up, with tech leaders like Bill Gates warning that addressing risks from A.I. should be “the world’s top priority.”
That urgency could shape talks between President Trump and China’s leader, Xi Jinping, that are expected on Sept. 24 in Washington, where the two sides are expected to discuss A.I. security as well as trade and other issues.
New risks seem to surface almost daily. Anthropic, the company behind Claude and other A.I. models, said in a report on Thursday that it had disrupted plots by scientists to use its models for research that could have helped develop biological weapons.
Anthropic also described attempts by a Chinese arms manufacturer to use Claude to refine a proposal for an anti-torpedo weapons system for the Chinese military. And it documented instances in which Chinese users tried to use Claude to build surveillance operations targeting foreign governments, Uyghurs in Syria and religious leaders. (Asked about the report, Mao Ning, a spokeswoman for China’s Foreign Ministry, said Friday that Beijing “opposed the distortion of facts and attempts to attack or smear China.”)
In the WeChat hack, the researchers who developed the tool were with Calif, a security company based in Palo Alto, Calif., that says it makes such tools not to sell them but to bolster cyberdefense.
They showed that the tool, which they named WeWorm, could hijack a WeChat user’s account, call their contacts and then spread from phone to phone without anyone ever answering a call. Calif said it took the company a little more than a week to build the bug and that it had disclosed the flaw to the White House and Tencent, the owner of WeChat.
In one sense, the news of WeWorm should make Beijing more eager to work with Washington to manage the risks posed by A.I., according to Kyle Chan, a fellow at the Brookings Institution focusing on Chinese technology and industrial policy.
“It hints at the massive potential for other actors, nation-states or nonstate actors, to use A.I. to attack China’s digital infrastructure,” Mr. Chan said. “The stakes have gotten much higher as A.I. systems make leaps in cyber capabilities.”
But deep mutual suspicion between China and the United States could undercut talks. And the revelation of weapons like WeWorm, even as a simulation, could simply accelerate competition over who will wield the world’s most powerful A.I. models.
China, now acutely aware of the vulnerabilities in its digital infrastructure, will most likely want to beef up the defense capabilities of domestic A.I. systems to better spot and fix security flaws, said Mr. Zhao, of Fudan University.
The problem is that the U.S. side may then see those systems as potential tools to attack U.S. companies and institutions, pushing the U.S. to strengthen its own cyber defenses. “The boundary between defense and offense gets increasingly blurred,” Mr. Zhao said. “We can call this the A.I. security dilemma.”
WeWorm and Anthropic’s report about abuses of its models are yet more evidence that the world’s two largest A.I. powers need a way to share information about bad actors or to clarify their intentions, experts say.
At the same time, few expect the summit to result in limits on the technology itself. Neither government is likely to agree to anything that restricts its own development of A.I. capabilities. But experts say there is value in at least establishing a channel of communication in the event of a crisis.
“This is an alarm, a wake-up call,” said Jiang Tianjiao, an associate professor at Fudan University focusing on emerging technologies, referring to WeWorm.
“Everyone needs to sit down and discuss how A.I. raises safety and security risks that are very different from traditional issues, and whether we need new frameworks of cooperation,” he said.
Even as China embraces A.I. as a strategic technology critical to its economy, officials have also begun to raise concerns about its risks.
Chinese regulators have warned local governments and companies about using the A.I. assistant OpenClaw and in May released guidelines for how to regulate the use of A.I. agents. Officials are also working on more than a dozen new national standards to strengthen cybersecurity in the age of A.I. Mr. Xi, speaking in July at an A.I. conference, said countries should work together to “ensure that A.I. is always under human control.”
Zhao Lei, director of the research office at the Central Party School, which trains Chinese Communist Party officials, warned in an essay published on WeChat on Thursday that extremist forces could leverage A.I. to design “highly transmissible and virulent pathogens” or to optimize weapons. This threat, he said, would “trigger a new kind of existential crisis for humanity.”
Last year, China’s cybersecurity authorities warned about the risk that extremist groups or others could use A.I. to learn how to design and build “nuclear, biological, chemical and missile weapons.”
Another issue hanging over A.I. safety talks with Washington is how transparent both sides would be. It is unclear whether Chinese A.I. firms would approach an American company if they found a flaw in their system, in the way that Calif described. Chinese firms are required to report such discoveries first to China’s Ministry of Industry and Information Technology.
Pei Wang, a cybersecurity expert, said that Chinese companies would approach such disclosures “with caution,” given the risk that they could be politicized amid the intense tech competition between the two countries.
China has long accused Washington of trying to hold back China’s technological progress with sanctions on Chinese companies, export controls cutting off the country’s access to advanced technology and tariffs on Chinese goods.
The WeWorm vulnerability sends a signal to Beijing about American A.I. capabilities at the same time that U.S. officials have signaled new willingness to attack Chinese companies. On Tuesday, the deputy C.I.A. director, Michael Ellis, said that Chinese companies were a legitimate target for espionage in areas like artificial intelligence, semiconductors and biotechnology.
Some Chinese scholars have suggested that American and Chinese officials could potentially discuss voluntary “negative lists” of unacceptable A.I.-enabled cyber activities or identify categories of critical infrastructure that are protected.
Chinese officials have not publicly acknowledged the disclosure of the WeChat hacking tool, perhaps in an effort not to add more tensions between the two countries before the summit. Ms. Mao, the foreign ministry spokeswoman, was asked about WeWorm at a news conference on Wednesday. She said she “was not familiar” with the issue.
For Tencent, the hack is likely to draw intense scrutiny by Chinese regulators, said Xiaomeng Lu, who researches U.S.-China tech competition at Eurasia Group.
“I wouldn’t be surprised if after this event agencies like the Cyber Administration of China invite some Tencent people over for tea and ask them exactly what happened and what’s the chance of something happening again,” Ms. Lu said.
Dustin Volz contributed reporting from Washington and Steve Lohr contributed reporting from Seoul.
Discussion about this post