Today, October 30, was the day Meta's updated Commercial Terms were due to take effect. Under them, a business that uses Facebook, Instagram, WhatsApp or Threads gives Meta permission to crawl the websites "associated with" its products or services. After a week of pushback from publishers, Meta moved the date to November 20. If your business has a Facebook Page, an Instagram profile or a WhatsApp Business account and also a website, you have about three weeks to decide what Meta's crawlers may read. Here is what changed, what is still unclear, and a practical checklist. Every fact is linked.
1. What Meta changed, and when
Meta told business users about the update in early October, in a blog post and inside Meta Business Suite, Nieman Lab reported on October 15. The new terms cover business accounts that manage Pages and groups, run ads, send marketing messages or use Meta's business dashboard. Nieman Lab asked three legal experts to review them, and all three said the wording, as first published, could be used to argue that Meta may scrape news sites and use that content to train its AI products.
- Mid-October: Meta removed the word "train" and limited the stated purpose to "advertising and business products" (Digiday, Search Engine Journal).
- October 22: publisher trade bodies around the world sent letters opposing the terms and urged lawmakers to act (Digiday).
- October 23: Meta said it would move the effective date from October 30 to November 20 (Digiday). Its Business Help Centre now reads: "From November 20, 2026, updated versions of the Meta Commercial Terms and the Self-Serve Ad Terms will take effect."
So nothing switched on today. The terms are paused, not withdrawn.
2. What the clause actually says
In the updated Commercial Terms Meta has published, a business authorises Meta "to use automated software (such as web scrapers, bots, robots, spiders, crawlers, and user agents) and create test credentials to access, index, cache, analyze, collect, retrieve, or crawl the domains, web pages, or URLs" that are associated with its products or services, or that are made available through its own or users' use of Meta's products. Meta may also "periodically pull updates" from those pages.
The stated purpose is to "provide, develop, improve, research, and maintain advertising and business products", plus "the integrity and security" of Meta's products. Two details matter for small businesses:
- You do not have to run ads to be covered. Meta's ad terms already let it crawl advertisers' websites; the October version reaches businesses that do not buy ads, such as anyone simply managing a Facebook Page (Search Engine Journal, Search Engine Journal).
- robots.txt is not in the clause. The new wording does not mention robots.txt, the standard file websites use to tell crawlers what to skip (Search Engine Journal). Meta says it still respects it (see below).
3. Meta's view, and the critics'
Meta's side. Meta says it is merging overlapping advertiser and business terms and that the update does not change how it crawls content (Digiday). Spokesperson Andy Stone wrote on X that the change is meant to help Meta pull accurate information about products for sale on other websites for ad and content recommendations, and that it is "not about scraping news sites" (Search Engine Journal). Meta also told Nieman Lab that the updates do not give it "new rights that supersede" robots.txt preferences (Search Engine Journal). Announcing the delay, a Meta spokesperson said content owners "will still be able to use robots.txt to signal preferences" (Digiday).
The critics' side. The legal experts were not reassured by the edit. Attorney Matt Topic said "develop" is broad enough to include "train"; Cornell law professor James Grimmelmann called the purpose wording "still broad enough to drive a truck through"; and Wisconsin law professor Benjamin Sobel noted that following robots.txt is voluntary and gives a site no enforceable legal right (Search Engine Journal). The News/Media Alliance, which represents more than 2,000 publishers, asked Mark Zuckerberg to withdraw the terms, and the World Association of News Publishers (WAN-IFRA) and several European bodies also objected (Digiday). Digiday notes that publishers currently cannot opt out of the terms without giving up Meta's business products.
4. Meta's crawlers, in plain words
Meta does not use one crawler. Its own crawler documentation lists several, each with its own name (its "user agent"), so a website can treat each one differently:
| Crawler | What Meta says it does | Does robots.txt stop it? |
|---|---|---|
| FacebookExternalHit | Reads a page shared on Facebook, Instagram or Messenger to show its title, description and thumbnail | Yes, but it "might bypass" robots.txt for security or integrity checks |
| Meta-ExternalAgent | Use cases "such as training foundation AI models or improving products by indexing content directly" | Yes, per Meta |
| Meta-WebIndexer | Improves Meta AI search results; allowing it helps Meta AI cite and link to your content | Yes, per Meta |
| Meta-ExternalAds | Improving advertising and other business-related products | Yes, per Meta |
| Meta-ExternalFetcher | Fetches individual links at a user's request, including AI that navigates websites to complete tasks | Not always: Meta says it "may bypass robots.txt" |
Search Engine Journal makes the same point: because Meta lists one crawler for AI training and another for Meta AI search, a business can block one and allow the other. What remains unclear is whether those robots.txt choices limit what a business grants when it accepts the terms.
5. A checklist for the next three weeks
For many Indian shops, D2C brands and creators, Instagram and WhatsApp are where customers find them, so leaving is not a real option. The website is where you have control. In our view, this is a sensible order of work:
- List your "associated" websites. Start with every site you have linked from your Facebook Page, Instagram profile, WhatsApp Business profile, product catalogue or ads. Those are the obvious candidates.
- Decide crawler by crawler. A common choice is to keep link previews and Meta AI citations but say no to AI training. Using the user agent names from Meta's documentation, that looks like this in robots.txt:
Leave FacebookExternalHit allowed if you want your links to show a title and image when shared. If you run Meta ads, test before blocking Meta-ExternalAds, because Meta does not spell out what blocking it changes for your campaigns.User-agent: meta-externalagent Disallow: / User-agent: meta-webindexer Allow: / - Treat robots.txt as a request, not a lock. Meta says two of its crawlers may bypass it, and the legal experts say it is not enforceable. Dealer price lists, customer files and drafts belong behind a login, not at a public link nobody is "supposed" to find.
- Check your server logs. Search your hosting logs for "meta-externalagent", "meta-webindexer" and "facebookexternalhit" to see which ones already visit and which pages they read.
- Read the final text before November 20. Meta has already changed the preview once (Search Engine Journal). Look at the updated terms again in mid-November. If your content is your product, as it is for publishers, course creators and research firms, get legal advice.
The bottom line
Meta's new business terms did not start today, but they are coming on November 20, and they cover far more businesses than just news sites. You cannot negotiate the terms, but you can decide what each Meta crawler may read on your own website. Spend ten minutes on your robots.txt this month, and keep anything private behind a login. AI companies are changing the rules for Indian businesses fast; for another example, read our guide to ChatGPT Ads in India.
FAQ
Did Meta's new business terms start on October 30, 2026?
No. Meta moved the effective date of its updated Commercial Terms and Self-Serve Ad Terms to November 20, 2026, saying it needed more time to explain the changes.
Which businesses do Meta's new terms cover?
Businesses that use Meta's products for business purposes, such as managing a Facebook Page, an Instagram business profile, WhatsApp or Threads business accounts, running ads or sending marketing messages. You do not need to run ads to be covered.
Can Meta use my website to train its AI?
Meta removed the word "train" from the clause and says crawling is for advertising and business products. Some legal experts say the word "develop" may still cover training. Meta's documentation says its Meta-ExternalAgent crawler is used for purposes such as training foundation AI models, and robots.txt can block it.
How do I block Meta's AI training crawler?
Add "User-agent: meta-externalagent" followed by "Disallow: /" to the robots.txt file at the root of your website. Meta says it respects robots.txt, though robots.txt is a request and not a legal right.
Will blocking Meta's crawlers break my link previews?
Only if you block FacebookExternalHit, the crawler that reads shared links to show a title, description and thumbnail. Blocking only Meta-ExternalAgent does not touch it.
Sources
- Nieman Lab: Meta's new commercial terms will allow the scraping of news sites, legal experts say (Oct 15, 2026)
- Search Engine Journal: Meta's New Terms Expand Crawling Rights Beyond Advertisers (Oct 2026)
- Search Engine Journal: Meta Drops Training From Its Crawl Clause And Keeps The Crawl (Oct 2026)
- Digiday: Meta delays rollout of updated commercial terms amid publishers' collective pushback (Oct 26, 2026)
- Meta Business Help Centre: Streamlining Meta's Commercial Terms (effective November 20, 2026)
- Meta: Meta Commercial Terms, updated version effective November 20, 2026
- Meta for Developers: Meta Web Crawlers
Quotes from Meta's terms and crawler documentation were checked on Meta's own pages on October 30, 2026; Meta may change them before November 20. Statements by Meta's spokespeople and the legal experts are as reported by the outlets named. This article is general information, not legal advice. The cover image is an illustration.