Meta removes terms language that critics said allowed news-site scraping for AI

The company deleted a clause that had stated Meta may use any content from "such domains, web pages, or URLs" to "provide, develop, improve, research, train, and maintain the integrity of" Meta products, services and features under the commercial terms and supplemental terms. The language no longer appears on the commercial terms page, according to The Verge.

Many observers, including Neiman Lab and James Grimmelmann, a professor of digital and information law at Cornell, interpreted the clause as an attempt to give Meta the right to use content scraped from websites to train its AI products for free and to override blocking instructions in a site's robots.txt file. The clause's removal came after those interpretations drew criticism.

Meta executives initially disputed that reading. Andy Stone, Meta's vice president of communications, and spokesperson Daniel Roberts said, "There is no change to how Meta uses web crawlers; any suggestion to the contrary is false." The company then updated its terms and issued a new statement that removed the controversial language entirely.

Roberts said in a follow-up statement to The Verge: "We've heard you and want to clarify: the crawling terms we consolidated only applied to our ads and business products, and we've clarified that in the terms themselves. It does not change how Meta uses web crawlers. As always, content owners can use robots.txt to signal their preferences. This is not about scraping news sites, and these updates don't give us new rights that supersede those preferences."

The episode centered on the intersection of AI training data and website owners' ability to control automated access. Robots.txt files are used by sites to signal whether crawlers may access their content. Meta's revised terms now state the consolidated crawling provisions apply to ads and business products, according to Roberts' statement. The company said the changes do not alter its use of web crawlers or grant rights that override robots.txt preferences.

The Verge reported that Meta was updating its terms again to remove the language. The first response from Stone and Roberts rejected the interpretation that the company had expanded its scraping rights. The revised language and statement go further by deleting the provision that had prompted the criticism.