How to Check Which Sites Block AI Crawlers: GPTBot, ClaudeBot and Google-Extended
Most writing about AI crawlers tells site owners how to block them. This post flips the question: how do you check, programmatically and at scale, which sites already block AI crawlers like GPTBot, ClaudeBot or Google-Extended, and what does that mean for their visibility in AI-generated answers?