Claude-SearchBot: user agent, IP verification and robots.txt
Claude-SearchBot is operated by Anthropic. It is classified here as Search (Search index). Verification method: Published CIDR list.
Exact user-agent string
Anthropic's crawler documentation names the robots.txt token but does not publish a full example user-agent string.
robots.txt token: Claude-SearchBot. The user-agent string is self-asserted and can be
forged by anyone. It is a claim, not proof.
How to verify Claude-SearchBot is genuine
Match the source IP against Anthropic's published CIDR list: https://claude.com/crawling/bots.json
20 ranges published, as of 2026-05-01. Pulled live at build time, never hardcoded.
How to block or allow Claude-SearchBot
User-agent: Claude-SearchBot
Disallow: /Replace Disallow with Allow to permit it explicitly.
What blocking Claude-SearchBot costs you
Disabling Claude-SearchBot prevents Anthropic indexing your content for search result quality. This is the Anthropic equivalent of blocking OAI-SearchBot: it is the visibility token, not the training token.
What is specific to Claude-SearchBot
Claude-SearchBot is the newest of Anthropic's three tokens and the one most often missing from robots.txt files that were written when ClaudeBot was the only name in circulation. A site that added a ClaudeBot disallow in 2024 to opt out of training did not opt out of search indexing, because tokens do not inherit. Anthropic states each setting is independent. It shares the single bots.json IP list with the other two Claude robots, so IP verification cannot separate it from ClaudeBot traffic.
Verify a request claiming to be Claude-SearchBot
Check a Claude-SearchBot request against Anthropic's published ranges
Source
Primary documentation: https://support.anthropic.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler. Last verified 2026-08-02.
Other Anthropic crawlers
- ClaudeBot - AI training crawler
- Claude-User - User-triggered fetch