Google-Extended: user agent, IP verification and robots.txt

Google-Extended is operated by Google. It is classified here as Control token (Control token, not a crawler). Verification method: None available.

Exact user-agent string

Google states Google-Extended has no separate HTTP user-agent string. Crawling is done with existing Google user agents; the token exists only as a robots.txt control.

robots.txt token: Google-Extended. The user-agent string is self-asserted and can be forged by anyone. It is a claim, not proof.

How to verify Google-Extended is genuine

Google publishes no IP range file and no reverse DNS method for Google-Extended. There is no way for anyone, including us, to prove a request claiming this identity is genuine.

How to block or allow Google-Extended

User-agent: Google-Extended
Disallow: /

Replace Disallow with Allow to permit it explicitly.

What blocking Google-Extended costs you

Disallowing Google-Extended withdraws your content from training future Gemini models and from grounding, where Google supplies content from its Search index to the model at prompt time. Your Search rankings are untouched. Your presence inside Gemini answers is what you are giving up.

What is specific to Google-Extended

Google-Extended will never appear in your access logs, and that is the single most misunderstood fact about it. It has no user-agent string of its own. Google crawls with the existing Googlebot agents and then applies your Google-Extended preference afterwards, as a use-time control rather than a fetch-time one. This makes it unverifiable by design: there is no request to check, no IP to match, and any log line claiming to be Google-Extended is fabricated. It is also the only token in this set that governs grounding as well as training, so blocking it affects answers Gemini generates today, not just models trained tomorrow.

Verify a request claiming to be Google-Extended

Check a Google-Extended request against Google's published ranges

Source

Primary documentation: https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers. Last verified 2026-08-02.

Other Google crawlers

All crawlers and the full comparison table