The SPUR Coalition, of which the Independent Media Association is a member, has recommended that publishers who don’t want their content used to train AI models adopt a new opt-out control backed by OpenAI, Google, Microsoft and Anthropic.
The proposed standard would allow publishers to set out their AI training preferences once, through a single shared signal in their website’s robots.txt file, rather than writing separate instructions for each AI crawler. Anthropic has already adopted the standard, while OpenAI plans to do so in the coming months and Google and Microsoft expect to support it in 2027. Meta has said it supports the idea of an industry-wide signal, but has not confirmed whether it will honour this one.
To opt out of AI training across a whole site, publishers can add these lines to their robots.txt file:
User-Agent: __usage_ai-training
Disallow: /
Google and Microsoft have said that opting out this way won’t affect how a site appears in conventional search results.
However, SPUR has noted that the control has significant limitations, despite recommending that publishers adopt it. The coalition said it would only be effective if every AI company respects it, including the third-party scrapers that collect content on their behalf. It also applies only to AI training, and does not cover the retrieval, grounding and AI-generated summaries through which AI tools increasingly make use of news content.
The coalition also raised concerns about how the proposal was developed. According to SPUR, it was drafted by a member of Anthropic’s technical staff and has been advanced by a small group of frontier AI labs. Although it is being taken forward through the Internet Engineering Task Force’s AI Preferences working group, SPUR said it has not yet been through the open review and consensus process that standards would normally be expected to follow.
For Independent Media Association members, adding the signal is a quick, practical way to put their training preferences on record. Meanwhile, SPUR will keep pressing AI companies for broader measures that would give publishers real control over how their journalism is used in AI systems.