Meta has delayed the introduction of new commercial terms that legal experts warn could allow it to scrape content from news publishers’ websites, giving publishers until 20 November to take steps to protect their content.
Under the updated terms, which were originally due to come into force on 30 October, businesses with a Facebook, Instagram or WhatsApp business account authorise Meta to use crawlers and other automated tools on websites “associated with” that account. The terms state that the content collected can be used to develop Meta’s advertising and business products. According to Nieman Lab, almost every news publisher holds at least one Meta business account, and legal experts consulted by the outlet warned that the terms could allow Meta to train its AI on publishers’ content without a licensing agreement. Meta has since removed the word “train” from the clause, although lawyers have argued that the remaining language is broad enough to cover AI development. Editor & Publisher reports that Meta has now delayed the start date to 20 November to allow time for consultation with publishers and advertisers.
Meta has insisted the update is not about scraping news sites, and says website owners can still control its crawlers through their robots.txt file. The News Media Association has called the move a “shameless landgrab”, while the campaign group Movement for an Open Web is preparing a challenge through the European Commission and the Competition and Markets Authority.
For Independent Media Association members, the most immediate step is to tell Meta’s crawlers not to collect your content for AI. Meta runs several different bots, and blocking all of them can stop your articles showing a headline and image when they are shared on Facebook. Members should therefore block the AI crawlers while keeping the bot that builds link previews. Adding the following lines to your website’s robots.txt file will do this:
User-agent: meta-externalagent
Disallow: /
User-agent: meta-externalads
Disallow: /
User-agent: facebookexternalhit
Allow: /
Meta says its crawlers can take up to 24 hours to notice changes to robots.txt. After making the change, members should check that links shared on Facebook still show previews as normal.
However, robots.txt is not legally binding, and Meta’s own documentation states that some of its crawlers may not follow it in certain cases, such as security checks. Industry experts therefore recommend that publishers using a content delivery network such as Cloudflare also switch on its AI crawler blocking tools, which stop unapproved bots at the server level. Movement for an Open Web has also advised publishers to update their website terms and conditions to state that scraping is prohibited and will incur a fee.
Some publishers have gone further and are considering closing their Meta business accounts altogether. For most independent outlets, which still rely on Facebook and Instagram to reach audiences, the measures above are a more practical first line of defence.
To read Press Gazette’s full report on Meta’s new terms and the advice for publishers, click here.