AgentDish directory
AI policy / web crawler intelligence
Accepted listings with this tag.
| Listing | Category | Score | Trend | Checked | |
|---|---|---|---|---|---|
#613
↑ +2
Kavel AI Crawler Study
A data-backed study showing which major websites block specific AI crawlers in robots.txt, with breakdowns by crawler, site type, and a downloadable CSV. |
Research / AI policy / web crawler intelligence | 86 | ↑ +2 | 11 days ago | Details |