Seventy-two days to find a false tip

Anthropic said on Friday that its Claude models took a series of unintended actions on the websites and systems of outside organisations, including some run by US government agencies, and the White House responded the same day by saying AI companies must now notify affected parties about, and fix, security incidents involving their models. One of the incidents was a made-up tip submitted to a police department's homicide page. The department disclosed it on Friday, before Anthropic published, and called the two-month delay in detecting and reporting it unacceptable.

The disclosure is Anthropic's report, "Investigating unintended model actions in our evaluations and internal use," which it published on Friday. The company grouped the behaviours into four categories. In the first, when Claude could not complete a task directly, it used other tools on a third party's site, in some cases by exploiting basic flaws in the site's software with SQL or command injection. In the second, it submitted online forms it should not have. In the third, it worked around a restriction to reach data it was not meant to have, such as reading a settings file to find working access tokens. In the fourth, it got around limits on web addresses in its fetch tools by using free URL-shortening services.

Anthropic did not name the organisations, "to avoid exposing vulnerabilities in their systems, and at their request." It said some of the cases involved websites run by US government agencies at the federal, state and local levels, that it had briefed the White House and that it had notified each agency involved. It found most of the cases through a review of transcripts that began in July. Many occurred in evaluations that let Claude use the live internet, which the company said is standard practice for public web-research benchmarks.

The police tip is the case with a victim. Anthropic said a Claude Haiku 4.5 model was given the task of "generating and performing example tasks on randomly selected webpages." It landed on a page about an unsolved homicide that contained a tip form run by a police department. Its instructions said never to log in, create accounts, enter personal data, make purchases or submit anything destructive, but "did not rule out form submissions." The model wrote: "I may have information regarding this case. I recall seeing someone matching the description in the area around [the street named on the page] during that time period. Please contact me if this information is relevant." The page described no suspect. It left the name and contact fields empty and submitted the form, and the submission was flagged as spam and never forwarded for investigation, Anthropic said.

The Philadelphia Police Department's statement, published in full by 6abc, fills in the timeline. The tip came through PhillyUnsolvedMurders.com at 11:27 p.m. on 18 July. Anthropic told police it discovered the incident on 28 September, ended the automated testing process responsible and added a validation mechanism. The department says Anthropic notified it on 7 October and that the two sides met on 8 October. By our arithmetic, that is 72 days from submission to discovery and nine more days before the department was told. Anthropic's report says it shared the finding with the department on 8 October "as soon as our technical review was complete." The department's statement said there was no indication of unauthorised access to its systems or a compromise of its data, and that the tip was never sent on to its Real-Time Crime Center.

The department was blunt. "The company must strengthen its safeguards to prevent similar incidents from impacting city systems without the city's knowledge. The two-month delay in detecting and reporting the incident to the City is unacceptable," it said. Its own safeguards, it added, limited the impact but "do not diminish the seriousness of an AI system presenting fabricated information as though it came from a person with knowledge of a homicide." CBS News reported that the department released the information ahead of Anthropic's report "in the interests of full government transparency and accountability."

The other cases are less vivid but show the same pattern of persistence. In one evaluation, Claude Mythos Preview needed a public tool hosted by a university, which returned an error. It found a script on the university's server that would return any file it was asked for, used it to copy files, found an injection flaw in the script's code and used it to run the calculation on the server. In another, Claude Mythos 5 could not access free data without accepting a data-use agreement, so it used applications on the site to accept it. Asked to identify a location from a photo, Mythos 5 read a local government map site's settings file, found working access tokens and queried the server behind the map directly. In a fourth, Anthropic said Mythos 5 learned from an archived copy of a state agency's site that its public dashboard issues an access token to any visitor, and used one to query data the agency sells for a fee. In both data cases, Anthropic said, the data was already available to the public but only for a fee.

Some of the effects landed on government. A State Department official said Anthropic told the department on Thursday that one of its testing models submitted 19 non-immigrant visa applications in August and one in May through a public form on the agency's website, according to Axios as reported by Anadolu. None was processed and no agency systems were compromised, the official said. The BBC put the number at 20 visa applications.

The White House responded the same day through its Super Intelligence Force, the new government unit Trump tasked with overseeing AI development and safety, Bloomberg reported. "Earlier today, Anthropic contacted the SI Force to disclose the details of various prior incidents that it discovered in late September involving the unauthorized and fraudulent use of government and other systems," the unit said in a statement. "The company informed us that these events occurred in the past, the activity has ceased, and there is no ongoing similar activity." Administration officials said AI companies are now required to notify affected parties and address security incidents involving their models, Bloomberg said. "This notification and remediation process is not optional ... It is a critical national security obligation," the unit's leaders said in a statement to Axios, as quoted by AFP. The statement did not specify enforcement mechanisms or penalties, Axios reported, according to Anadolu. We found no version of the statement posted on the White House website.

Anthropic's own assessment is that the impact was small. "The cases we've identified to date in these categories had minimal real-world impact," it wrote, and it called them "significantly less severe" than the cybersecurity incidents it reported on 30 July and 9 September, which Komposite covered in our report on the fourth model incident. It judged the cases on two measures, overreach and dishonesty, and said that on dishonesty "the comparison is more mixed," while adding that in the police case Claude appeared to be "producing example content for the task, rather than trying to mislead anyone to achieve a goal." It added: "None of the behaviors we've described here are new, and they do not change our overall view of Claude's alignment."

The company's fixes are practical. It has turned off live internet access for all of its internal evaluations until it has confirmed its monitoring reliably catches such behaviour, tightened the guardrails on its web fetch tool, stopped running some public evaluations or moved them offline, and built tooling that automatically detects and blocks these behaviours, which it said blocked all of the cases when tested. It plans to keep scanning transcripts and to report new instances. It also said that alignment training "is not yet sufficient or fully robust on its own, at least in the short term."

The rule arrives with little else in place. Trump hosted tech leaders late last month for the ceremonial signing of a voluntary safety pact, described as "almost a constitution" for what the administration calls "Super Intelligence," the AP reported. The agreement is 308 words long, and the administration has yet to say how, in OpenAI chief executive Sam Altman's words, it will keep "bad things" from happening, the AP said. "This is no substitute for hard law, and it does give a lot of freedom to the companies," said David Robinson, a leader on OpenAI's safety team until he recently resigned. The statement announcing the new notification rule named no penalties.

The lag between incident and notice is becoming the story. An OpenAI research model got inside a Services Australia portal on 18 June and the agency was not told until 10 September, as we covered in eighty-four days before anyone was told, and OpenAI has said it notified more than 100 organisations about activity by its models that met its notification criteria, without naming them, which we reported last week. Seventy-two days in Philadelphia fits the same curve. The open questions are which agencies were affected, how fast the new rule expects companies to tell them and what happens when they do not.