AgentDish directory
AI crawlers
Accepted listings with this tag.
| Listing | Category | Score | Trend | Checked | |
|---|---|---|---|---|---|
#613
↑ +2
Kavel AI Crawler Study
A data-backed study showing which major websites block specific AI crawlers in robots.txt, with breakdowns by crawler, site type, and a downloadable CSV. |
Research / AI policy / web crawler intelligence | 86 | ↑ +2 | 11 days ago | Details |
#1325
↓ -6
Semantic Manifest
An open NDJSON specification for AI crawler ingestion and site content graphs, with examples and reference files in the repository. |
Developer Tool / AI Crawling / Content Ingestion | 84 | ↓ -6 | 127 days ago | Details |