Business#Web scraping#Model training
Meta deletes the AI-training sentence from its business terms, and keeps everything that lets it crawl
Meta deleted the sentence that let it train on business-site content before Oct. 30, keeping the crawl authorization, test credentials and "develop."
The Meta Commercial Terms are the contract every account using Facebook, Instagram, WhatsApp, Messenger or Business Suite for a commercial purpose has to accept, and the preview page says they take effect on October 30, 2026. On Friday, October 16, the standalone sentence that let Meta use content scraped from business sites to “train” its products disappeared from that contract; the permitted uses moved into the sentence before it and narrowed to “advertising and business products.” The edit is checkable word for word. A Wayback capture from 16:20 UTC on October 16 still has the old sentence, and one from 03:19 UTC on October 17 does not. The CDX index lists nine captures for this page between September 1 and October 18, and all nine contain the authorization clause: five before the change, four after. The live page matches the new version: the word train appears zero times, advertising and business products once.
Key points
- What was deleted is a whole sentence: “Meta may use any content from such domains, web pages, or URLs to provide, develop, improve, research, train, and maintain the integrity of, the Meta Company Products, services, and features provided under these Commercial Terms and any other applicable supplemental terms.”
- The crawling permission is untouched: access, index, cache, analyze, collect, retrieve, crawl, and “create test credentials.” So is the verb “develop,” which now points at advertising and business products.
- The terms never mention robots.txt. Meta says site owners’ preferences still apply; legal scholars say compliance was voluntary to begin with. The October 30 date stands, and business accounts have no opt-out.
What the edit changed
The old paragraph ran as two sentences. The first granted permission: Meta may use automated software — “web scrapers, bots, robots, spiders, crawlers, and user agents” — and create test credentials to access, index, cache, analyze, collect, retrieve or crawl domains, web pages or URLs associated with your products or services, and periodically pull updates. The second sentence, standing alone, said what the collected material could be used for, and train sat in that list of verbs.
The revision deletes that second sentence. The purposes now hang off the permission sentence with “in order to,” the verbs are provide, develop, improve, research, maintain, and the object changes from “the Meta Company Products, services, and features provided under these Commercial Terms” to “advertising and business products,” followed by a parenthetical definition. The closing phrase also shifts from “maintain the integrity of” to “maintain the integrity and security of.”
| Element | Before (16:20 UTC, Oct 16) | After (03:19 UTC, Oct 17 onward) |
|---|---|---|
| Purpose clause | Separate sentence: Meta may use any content … to provide, develop, improve, research, train, and maintain the integrity of, the Meta Company Products | Folded into the permission: … in order to provide, develop, improve, research, and maintain advertising and business products (…) |
| Scope | Meta products and services, plus features under these and any supplemental terms | Advertising and business products, defined as helping create and deliver ads and commercial content, tell users about your products, and connect you with interested users |
| Crawling grant | Verb list, “create test credentials,” “periodically pull updates” | Unchanged |
| Effective date and exit | October 30, 2026 | Same; disagreeing means stopping all business use |
The parenthetical deserves a close read. It defines advertising and business products as a set of tools around ad delivery and user connection, and says nothing about AI. A Meta spokesperson says the consolidated crawling terms “only applied to our ads and business products.” Whether generative AI inside an ad product counts is a question the text does not answer. The edit landed between 16:20 UTC on the 16th and 03:19 UTC on the 17th, an eleven-hour window that falls entirely on Friday in California.
What did not change
The permission’s object is intact: domains, web pages or URLs “associated with your products or services,” plus URLs “made available through your or users’ use of Meta Company Products and services.” That phrasing reaches past a company’s own site. James Grimmelmann of Cornell Law School told Nieman Lab the revision “does nothing to restrict the ‘associated with your products or services’ language that would let Meta scrape entire sites.”
“Create test credentials” also survives, meaning Meta may set up test logins for the sites it reaches. Grimmelmann said this would technically let Meta “sign up as a fake user” to get past a registration wall. Meta declined to answer whether the credentials could clear paywalls or paid content.
The exit has not moved either: “If you do not agree to the updated terms, you may stop all access or use of our Products for business or commercial purposes.” There is no opt-out button for business accounts before October 30.
The 30-day notice belongs to a different document. The preview text contains no promise to give 30 days’ warning; its own Updates section says Meta may update the terms from time to time and encourages users to check back. The 30-day commitment — “at least 30 days before we make changes to These Terms” — sits in Meta’s general Terms of Service, which the preview says apply alongside it. Treating it as part of this clause puts it in the wrong place.
One more detail worth pinning down: the commercial terms currently in force, effective March 6, 2024, contain neither “automated software” nor “create test credentials.” The crawling grant is new to this preview; it was not carried over from the old commercial terms. Meta describes the change as consolidating language scattered across agreements, and the existing Ad Creative Generative AI Terms do include automated retrieval — but that clause scopes it to “your use in connection with your ads,” which Nieman Lab summarized as using scraped content only to “optimize your ads.” That is much narrower than the new clause.
What others say
Nieman Lab’s Andrew Deck and Hanaa’ Tameez followed up on October 17 with three legal scholars. Matt Topic of Loevy & Loevy said “I think publishers should remain concerned,” because “business product” is not defined and “I think ‘develop’ is broad enough to include ‘train.’” Grimmelmann said the update “doesn’t address my concerns” and that the new purpose language is “still broad enough to drive a truck through.” Benjamin Sobel of the University of Wisconsin called robots.txt compliance “completely voluntary,” like a homeowner’s “Don’t look at my house” sign that creates no enforceable right.
Meta’s public line came in layers. On October 16 a spokesperson told Ars Technica that “there is no change to how Meta uses web crawlers; any suggestion to the contrary is false.” The statement after the edit read: “We’ve heard you and want to clarify: the crawling terms we consolidated only applied to our ads and business products, and we’ve clarified that in the terms themselves,” adding that “This is not about scraping news sites, and these updates don’t give us new rights that supersede those preferences.” Andy Stone disputed Nieman Lab’s reading to Search Engine Journal. At Nieman Lab’s publication time, Meta had not answered whether the test credentials could clear paywalls.
Publisher groups went further. Danielle Coffey of the News/Media Alliance called the terms “outrageous and probably illegal,” an “appalling, bad-faith policy,” and said “It cannot be allowed to be implemented on October 30.” Theo Bamber, chief executive of the UK News Media Association, called it a “shameless landgrab attempt” and “the worst kind of arrogant tech platform bullying.” Angela Mills Wade of the European Publishers Council said “burying such far-reaching claims in platform terms is no substitute for consent”; the EPC said it would write to EU executive vice-presidents Teresa Ribera and Henna Virkkunen asking for urgent scrutiny of the terms on copyright, contractual fairness and abuse of market power. INMA reported that some news organizations are weighing whether to stop publishing on Facebook and Instagram while their lawyers assess the revised wording.
The money is unquantified. Matt Topic’s line about “many millions of dollars in a licensing market” comes from Nieman Lab’s October 15 story; Ars Technica quotes him second-hand the next day, and Ashley Belanger reads the clause as a route around those licensing talks. Both outlets stop at “millions,” so no figure belongs in a retelling.
The story also started with a reader tip. Stephen Beale, founder of The Steampunk Explorer, contacted Nieman Lab after seeing a notice in his Business Suite dashboard, and three legal scholars then read the clause line by line. The first story ran on October 15, more than ten days after the earliest capture; Meta’s own notice never mentioned crawling.
Where robots.txt stands
Meta’s developer documentation lists five common crawler user agents, and their relationship to robots.txt is not uniform.
| User agent | Stated purpose | robots.txt |
|---|---|---|
meta-externalagent/1.1 | Training foundation AI models, or indexing content to improve products | Docs show a disallow rule; no bypass mentioned |
meta-webindexer/1.1 | Improving Meta AI search quality | Docs say allowing it helps Meta cite and link your content |
meta-externalads/1.1 | Improving advertising and other business products | Docs show a disallow rule; no bypass mentioned |
meta-externalfetcher/1.1 | Fetching single links at a user’s request, for agentic tasks | Docs say it “may bypass robots.txt rules” |
facebookexternalhit/1.1 | Fetching title, description and thumbnail for shared links | Docs say it “might bypass robots.txt” for security or integrity checks |
This table and the contract are two different instruments. The contract grants a contractual right; robots.txt expresses a preference. Meta’s docs tell site owners to disallow by user agent and warn that changes can take up to 24 hours to apply. The contract itself promises nothing about robots.txt: the assurance that the update grants no rights superseding preferences exists in a spokesperson’s statement, not in the clause. Sobel’s version is blunter — the preference can be expressed, and ignored all the same.
There is one loose end between the two. Meta’s documentation describes meta-externalagent as a crawler for training foundation models, while its spokesperson says the consolidated terms cover only advertising and business products. The first describes crawler behavior, the second describes a contractual permission, and both can be true while leaving a practical question open: under which clause is content collected by ExternalAgent and used for model training authorized?
Our take
For teams that run a site and also use a Meta business account to advertise or manage a page, October 30 is a real line: continued use counts as acceptance, and the only refusal is to stop using the products commercially. The first thing to inventory is which domains fall under “associated with your products or services,” because the clause also reaches URLs exposed through the commercial uses it lists — a product catalog, Business Suite, measurement services — often more than the person in charge assumes.
Keep writing robots.txt. It does real work against ExternalAgent, WebIndexer and ExternalAds, and the 24-hour cache belongs in any response plan. Treating it as a legal shield will fail: the clause does not mention it, Meta’s assurance stops at a statement, and Sobel’s point is that compliance was voluntary all along.
“Train was deleted, so training is off” is a conclusion the text cannot carry. Develop remains, “advertising and business products” has no exclusive definition, and Meta will not say whether test credentials reach paywalls. What is confirmed is a real narrowing, from all Meta products to advertising and business products. Where the new boundary sits is not confirmed. Meta’s claim that crawler behavior is unchanged, read next to its own documentation for ExternalAgent, leaves that question standing.
Two comparable moves ran this month. Reddit shut its RSS feeds and public API and closed the entrance; Wikimedia named OpenAI’s crawlers in public and put a company on the record. Meta wrote the permission into a contract businesses have to accept. All three point at the same fact: how much control a site has over crawlers depends on what the platform chooses to give up, not on what its robots.txt says.
Three things to watch: whether the terms take effect unchanged on October 30; whether the EPC’s complaint to the European Commission gets taken up; and whether any publisher actually closes its business accounts. The first two have dates. The third only has a wait.