AI crawler directory: who fetches your site, and what to allow
The working list of AI crawlers worth a robots.txt decision: who operates each one, what it feeds (training, search index, or live answers), and a decision framework, with this site's 15-crawler allowlist as the example.