New research adds to the growing number of reports of agents going rogue.
On Wednesday, an international group of researchers called Swarmchasers published a report claiming that they found a group of Chinese agents attempting to access the IT infrastructure of tech giant Alibaba. The researchers claim that the agents originated from a testing environment operated by Tencent Cloud, the Chinese tech and cloud service giant.
The attacks occurred between September 18 and October 8, and involved the agents attempting to answer questions about places in China using Amap, Alibaba's version of Google Maps. The bots attempted, and in some cases succeeded, at bypassing anti-bot protections, and, to answer questions, used Amap keys and security published the the web that had been leaked in a public repository as a means of getting information from systems not intended for public use.
After finding the fleet and reporting their findings at a Hackathon on October 4, the group found a way to make contact with these agents. In their conversations, though the agents were linked to Tencent, they often identified themselves as "Claude," potentially as a result of using similar instructions to what Anthropic uses to train Claude Code, or as a result of model distillation, a challenge that Anthropic has been attempting to rein in.
Notably, the researchers made the distinction that this incident differs from the infamous OpenAI-Hugging Face attack in July, as they found no evidence that the agents colluded with one another, labelling the group an agent "fleet" rather than a "swarm."
Additionally, the researchers wrote in a blog post that this incident is not unique to Tencent. Ethan Elasky, one of the report's authors, told Bloomberg that, no matter whether the agents are from a US lab or a Chinese one, the way that model development is happening will result in agents breaking free from sandboxes when given access to the internet.
"We believe this demonstrates that model alignment more generally is still an unsolved problem, and predict that if capabilities continue to advance as they have over the last few years, operators will find it increasingly difficult to restrain their models and guarantee safety," Swarmchasers wrote in the report.
This incident follows a litany of other cases of agents breaking free from their sandboxes, largely involving US labs, including OpenAI, Anthropic and Google. It also comes as researchers and experts urge the industry to put more resources into solving the problem of AI containment, with a coalition of hundreds of researchers on Tuesday publishing an open letter calling for an international standards body to create alignment best practices.
Our Deeper View
This incident highlights that the rogue agent problem may extend beyond the major US model labs, especially as the US and China race to develop increasingly powerful AI models. It also may reveal how model distillation attacks could bleed into the containment problem. Anthropic has been one of the loudest voices calling out these attacks, accusing Chinese labs of distilling and exfiltrating Claude's capabilities on more than one occasion. Additionally, Chinese model providers have also been accused of putting speed and development before safety, with a recent SemiAnalysis report finding that very few Chinese model releases in the last few years have been paired with safety reports. These incidents make an international safety standards body pushing for best practices and transparency all the more important. In short: An international problem calls for an international solution.