Meta-WebIndexer: user agent, IP verification and robots.txt
Meta-WebIndexer is operated by Meta. It is classified here as Search (Meta AI search index). Verification method: None available.
Exact user-agent string
meta-webindexer/1.1 (+/documentation/sharing/webmasters/web-crawlers)
robots.txt token: Meta-WebIndexer. The user-agent string is self-asserted and can be
forged by anyone. It is a claim, not proof.
How to verify Meta-WebIndexer is genuine
Meta publishes no IP range file and no reverse DNS method for Meta-WebIndexer. There is no way for anyone, including us, to prove a request claiming this identity is genuine.
How to block or allow Meta-WebIndexer
User-agent: Meta-WebIndexer
Disallow: /Replace Disallow with Allow to permit it explicitly.
What blocking Meta-WebIndexer costs you
Meta states that allowing Meta-WebIndexer helps it cite and link to your content in Meta AI's responses. Blocking it is a straight trade of crawl traffic for citations, and it is the only Meta token where Meta makes the citation benefit explicit.
What is specific to Meta-WebIndexer
Meta-WebIndexer is new. It does not appear in older versions of Meta's crawler documentation, which is why robots.txt files written around Meta-ExternalAgent do not cover it, and tokens do not inherit. If you opted out of Meta's AI crawling before mid 2026, you are almost certainly still being indexed for Meta AI search. Meta's page was last updated on 21 May 2026. Like every Meta crawler it has no published IP range file, so the token is the only control and the identity is unverifiable.
Verify a request claiming to be Meta-WebIndexer
Check a Meta-WebIndexer request against Meta's published ranges
Source
Primary documentation: https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/. Last verified 2026-08-02.
Other Meta crawlers
- Meta-ExternalAgent - AI training and indexing crawler
- Meta-ExternalFetcher - User-triggered fetch
- Meta-ExternalAds - Advertising and business products
- FacebookExternalHit - Link preview fetch