AI crawler source registry: what each operator actually publishes

https://developer.amazon.com/amazonbot/ip-addresses/

Client rendered only Amazon · covers Amazonbot · HTTP 200 · declared text/html, served text/html · 0 CIDRs parsed

zero CIDR blocks present without executing JavaScript

https://developer.amazon.com/amazonbot/live-ip-addresses/

Client rendered only Amazon · covers Amzn-User · HTTP 200 · declared text/html, served text/html · 0 CIDRs parsed

zero CIDR blocks present without executing JavaScript

https://docs.claude.com/claudebot.json

Wrong content type Anthropic · covers ClaudeBot, Claude-SearchBot, Claude-User · HTTP 200 · declared application/json, served text/html · 1 CIDR parsed

declared JSON, served text/html (735035 bytes). A verifier fetching this gets 200 and no ranges.

Widely cited as Anthropic's published range file. It returns HTTP 200 and serves the documentation site's HTML application shell, not JSON. A verifier that checks status codes rather than parsing the body treats this as a successful fetch of an empty range list and fails open: every request claiming to be ClaudeBot passes or every one fails, depending on which way the code errs. Anthropic's actual list is at claude.com/crawling/bots.json, named only in the body of a support article.

Documented location: https://claude.com/crawling/bots.json

https://support.anthropic.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler

Moved Anthropic · covers ClaudeBot, Claude-SearchBot, Claude-User · HTTP 200 · declared text/html, served text/html · 0 CIDRs parsed

redirects to https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler

https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers

Moved Google · covers Googlebot, GoogleOther, Googlebot News, GoogleOther-Image, GoogleOther-Video, Google-CloudVertexBot, Google-Extended · HTTP 200 · declared text/html, served text/html · 0 CIDRs parsed

redirects to https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers

https://developers.google.com/static/search/apis/ipranges/googlebot.json

Moved Google · covers Googlebot · HTTP 200 · declared application/json, served application/json · 169 CIDRs parsed

redirects to https://developers.google.com/static/crawling/ipranges/common-crawlers.json

The legacy path. It still serves valid JSON, so allowlists pinned here keep working, but Google's current documentation points at /static/crawling/ipranges/ and treats that as canonical. Anything pinned to the old path is depending on a URL Google no longer documents.

Documented location: https://developers.google.com/static/crawling/ipranges/common-crawlers.json

https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/

Moved Meta · covers Meta-ExternalAgent, Meta-ExternalFetcher, Meta-WebIndexer, Meta-ExternalAds, FacebookExternalHit · HTTP 200 · declared text/html, served text/html · 0 CIDRs parsed

redirects to https://developers.facebook.com/documentation/sharing/webmasters/web-crawlers

https://platform.openai.com/docs/bots

Moved OpenAI · covers GPTBot, OAI-SearchBot, ChatGPT-User, OAI-AdsBot · HTTP 200 · declared text/html, served text/html · 0 CIDRs parsed

redirects to https://developers.openai.com/api/docs/bots

https://docs.perplexity.ai/guides/bots

Moved Perplexity · covers PerplexityBot, Perplexity-User · HTTP 200 · declared text/html, served text/html · 0 CIDRs parsed

redirects to https://docs.perplexity.ai/docs/resources/perplexity-crawlers

https://www.perplexity.com/perplexity-user.json

Moved Perplexity · covers Perplexity-User · HTTP 200 · declared application/json, served application/json · 4 CIDRs parsed

redirects to https://www.perplexity.ai/perplexity-user.json

https://www.perplexity.com/perplexitybot.json

Moved Perplexity · covers PerplexityBot · HTTP 200 · declared application/json, served application/json · 8 CIDRs parsed

redirects to https://www.perplexity.ai/perplexitybot.json