Googlebot: user agent, IP verification and robots.txt
Googlebot is operated by Google. It is classified here as Search (Search index). Verification method: Published CIDR list and reverse DNS.
Exact user-agent string
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
robots.txt token: Googlebot. The user-agent string is self-asserted and can be
forged by anyone. It is a claim, not proof.
How to verify Googlebot is genuine
Match the source IP against Google's published CIDR list: https://developers.google.com/static/crawling/ipranges/common-crawlers.json
169 ranges published, as of 2026-07-31. Pulled live at build time, never hardcoded.
A reverse DNS path also exists. See https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests.
How to block or allow Googlebot
User-agent: Googlebot
Disallow: /Replace Disallow with Allow to permit it explicitly.
What blocking Googlebot costs you
Blocking Googlebot removes you from Google Search. There is no version of this that is a partial cost. It is also not the AI training lever: that is Google-Extended, which is a separate token.
What is specific to Googlebot
Googlebot is the only crawler here with two independent verification paths that both work. The published common-crawlers.json is one; the other is a reverse DNS mask, where a genuine Googlebot address resolves to a crawl-*.googlebot.com or geo-crawl-*.geo.googlebot.com hostname, and the forward lookup of that hostname returns the original IP. Google also moved these files: the canonical path is now under /static/crawling/ipranges/, and allowlists still pointing at the older /static/search/apis/ipranges/ location are pinned to a URL Google no longer treats as primary.
Verify a request claiming to be Googlebot
Check a Googlebot request against Google's published ranges
Source
Primary documentation: https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers. Last verified 2026-08-02.
Other Google crawlers
- GoogleOther - Internal fetch and analysis
- Googlebot News - Google News index
- GoogleOther-Image - Internal fetch, images
- GoogleOther-Video - Internal fetch, video
- Google-CloudVertexBot - Site-owner requested crawl
- Google-Extended - Control token, not a crawler