Open Data

The data behind the work.

Every study here ships its underlying data, primary-sourced and dated. Free to reuse under CC BY 4.0 with attribution to CrawlSpace Labs.

• Ghost Price: agent price-readability of 50 US retailers (2026 pilot)

  Whether a naive HTTP AI shopping agent could read a machine-readable product price at 50 of the largest US retailers, June 2026. Per-retailer outcome, blocking method, and evidence reference.

  results.json · CC BY 4.0 · 2026-06-20

• AI crawler robots.txt compliance registry

  A primary-sourced registry of AI crawler tokens and what each vendor states it respects in robots.txt, checked against the vendor's own published documentation. Verified and pending entries labeled.

  bots.json · CC BY 4.0 · 2026-06-15

• AI crawler operator IP-range and verification registry

  Known AI crawler operators, their published IP ranges, and verification methods (Web Bot Auth / reverse DNS), used to verify whether a request's user-agent matches its source.

  operators.json · CC BY 4.0 · 2026-06-20