Open Data
The data behind the work.
Every study here ships its underlying data, primary-sourced and dated. Free to reuse under CC BY 4.0 with attribution to CrawlSpace Labs.
- Ghost Price: agent price-readability of 50 US retailers (2026 pilot)
Whether a naive HTTP AI shopping agent could read a machine-readable product price at 50 of the largest US retailers, June 2026. Per-retailer outcome, blocking method, and evidence reference.
results.json · CC BY 4.0 · 2026-06-20
- AI crawler robots.txt compliance registry
A primary-sourced registry of AI crawler tokens and what each vendor states it respects in robots.txt, checked against the vendor's own published documentation. Verified and pending entries labeled.
bots.json · CC BY 4.0 · 2026-06-15
- AI crawler operator IP-range and verification registry
Known AI crawler operators, their published IP ranges, and verification methods (Web Bot Auth / reverse DNS), used to verify whether a request's user-agent matches its source.
operators.json · CC BY 4.0 · 2026-06-20