Despite repeated calls for intervention, the US government has largely resisted AI regulation. Now, rogue agents may have pushed the limits too far.

Last week, Anthropic published a report detailing unintended actions by Claude observed in internal evaluations and use cases. Though the AI lab chose not to name the organizations to avoid exposing their vulnerabilities, it noted the incident involved "websites run by U.S. government agencies at the federal, state, and local levels." And according to a recent report from The New York Times, these incidents triggered the White House's most aggressive AI regulatory response yet.

Broadly speaking, Anthropic boiled Claude's actions down to four categories: exploiting a basic software flaw to run commands on a server, submitting a sensitive form on a real website, working around a restriction to access gated data, and using URL-shortening services to bypass limits. The New York Times talked to people familiar with the matter to fill in the gaps:

• Anthropic's agents submitted 20 visa applications via a form available on the State Department's website. The applications were incomplete and not processed.
• The lab's technology also submitted a false homicide tip to the Philadelphia Police Department via the agency's website, dated July 18. The tip was flagged as spam and not investigated.

The White House AI task force, now called the Super Intelligence Force, was formed the week Anthropic alerted authorities of the breach and was briefed on the agents' actions on Friday. The agency demanded that AI companies immediately report rogue AI activity, which, as The New York Times highlighted, marks the most aggressive AI regulation the Trump administration has implemented yet, as its attempts to regulate AI have thus far largely been voluntary and "morally binding."

“We informed the company that we expect immediate and full transparency to the entities involved and the public,” The agency said in a statement to The New York Times. “We also expect that the company will immediately provide remediation services to the affected entities and any harmed Americans.”

In the blog post, Anthropic says it identified most of the cases through a review of transcripts that began in July, and says that these cases have minimal real-world impact — less severe from an alignment and security perspective than the cybersecurity incidents it previously reported on July 30 and September 9. Yet, in the wake of these findings, the company decides to disable live internet access across all internal evaluations, not just high-risk and cybersecurity evaluations.

Our Deeper View

The cycle of AI evaluations leading to AI agents going rogue and attacking third-party websites illustrates why evaluations, red teaming, regulation, and other precautions are necessary. However, it also highlights the need for alignment on what risks are worth taking when testing AI, and a protocol for reporting them. While the Super Intelligence Force could be part of the answer, its formation seems to have been haphazard, with a Politico report stating that both Silicon Valley and Capitol Hill were caught off guard by President Donald Trump's announcement on social media. Though the requirement to immediately report rogue agent incidents could help address the problem, it remains to be seen how effective this task force will be in keeping up with these models' capabilities.