Playground Sign in Start free
← All posts

How to Check Which Sites Block AI Crawlers: GPTBot, ClaudeBot and Google-Extended

Most writing about AI crawlers tells site owners how to block them. This post flips the question: how do you check, programmatically and at scale, which sites already block AI crawlers like GPTBot, ClaudeBot or Google-Extended, and what does that mean for their visibility in AI-generated answers? robots.txt lives at a predictable path when present, so you can audit who blocks AI crawlers across hundreds of domains with a short Python script and one API credit per domain. When the file is missing, you get a 404 you can log as "absent." By the end of this post you will have working code, a decision table for interpreting what you find, and a clear picture of how crawler access connects to whether a brand gets cited in ChatGPT, Gemini or Google AI Overviews.

Which AI Crawlers Are Affected and What robots.txt Directives Control Them

A web crawler is a bot that visits websites iteratively, follows links, and extracts content for further processing. AI companies use crawlers to build training datasets for large language models and to power retrieval-augmented answers. The three crawlers you will encounter most often in robots.txt files are GPTBot, ClaudeBot and Google-Extended.

The robots.txt syntax

robots.txt uses two core fields: User-agent names the crawler, and Disallow or Allow sets the access rule for that crawler. A Disallow: / blocks the entire site; Disallow: /private/ blocks only that path; Allow: /public/ carves out an exception. A wildcard User-agent: * applies to every crawler not named explicitly above it in the file.

Here is the standard syntax for blocking a few well-known crawlers, including Googlebot as a reference point:

User-agent: Googlebot
Disallow: /private/

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

The Googlebot entry above is a partial block, restricting only one path. The GPTBot and ClaudeBot entries are full blocks. Google-Extended is a content-use preference rather than a fetch block, which is explained below.

GPTBot

GPTBot is OpenAI's primary training crawler. Its user-agent string is GPTBot. OpenAI documents it and states that it respects robots.txt. OpenAI also operates OAI-SearchBot, which powers live web retrieval in ChatGPT, and ChatGPT-User, which fires when a user asks ChatGPT to browse a specific URL. Each of these three agents is separate, so a Disallow rule for GPTBot does not carry over to the others. Treat each one as a distinct policy decision.

ClaudeBot

ClaudeBot is Anthropic's training crawler, user-agent string ClaudeBot. Anthropic documents its robots.txt compliance and states that ClaudeBot respects Disallow rules. Anthropic also operates separate agents for search and user-triggered retrieval, so check Anthropic's documentation for the full list if you need granular control.

Google-Extended

Google-Extended is a special case. Google documents it as a content-use preference: it controls whether Google may use crawled content for AI training and its Gemini products, but it has no separate HTTP user-agent string at the network layer. A site can allow Googlebot to index pages for search while disallowing Google-Extended, meaning Google keeps the page in its search index but agrees not to use it as AI training material. You cannot block Google-Extended at the firewall the way you can block GPTBot by user-agent string.

Why blocking one bot does not block all AI engines

Each AI company runs multiple crawlers for different purposes: training, search retrieval, and user-triggered browsing. A Disallow rule for GPTBot blocks OpenAI's training crawler but leaves its retrieval and user-triggered agents unaffected. A wildcard User-agent: * block catches everything not named above it, but only if no named Allow rule overrides it for a specific crawler. Build a lookup table of the exact user-agent strings you care about before writing any code.

The enforcement gap

robots.txt is a convention with no technical enforcement mechanism. A crawler that ignores it faces no server-side block from the file itself, as published guidance on crawler policy notes. Compliance depends entirely on the crawler operator choosing to respect the rules. User-agent strings are also trivially fakeable: any bot can claim to be any crawler. An audit of robots.txt tells you what a site's declared policy is, not whether every crawler that visits actually obeys it. robots.txt is like a "staff only" sign on a door with no lock: most visitors respect it, but the sign itself stops no one.

This enforcement gap is also why some operators go beyond robots.txt and enforce private access with authentication and server-side authorization. robots.txt alone is not sufficient if you need a hard technical guarantee.

Takeaway: Build your agent lookup table first: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Google-Extended. A Disallow rule for one does not carry over to the others.

How to Fetch and Parse robots.txt at Scale with Python and Ujeebu's Scrape API

robots.txt lives at a predictable path: https://domain.com/robots.txt. Fetching it directly from your own machine works for a handful of domains, but across hundreds of sites your IP accumulates requests quickly and some servers will rate-limit or block it. Routing each request through a rotating proxy pool distributes the traffic across many exit IPs, reducing the likelihood that any single origin triggers a rate limit. For a deeper look at how proxy types affect scraping at scale, see our guide on residential proxies vs datacenter proxies and the broader discussion of IP classification and rate-limiting.

The Ujeebu Scrape API handles proxy rotation with a single parameter: proxy_type=rotating. Because robots.txt is plain text, you do not need JavaScript rendering, so js=false. That combination costs 1 credit per request.

Credit cost in practice: auditing 1,000 domains costs 1,000 credits. The free trial includes 5,000 credits with no credit card required, so you can audit up to 5,000 domains before spending anything. Failed or blocked requests are not charged, so a 404 or timeout does not consume a credit.

Here is the curl one-liner to fetch a single robots.txt:

curl -X GET 'https://api.ujeebu.com/scrape?url=https://scrape.li/robots.txt&js=false&proxy_type=rotating' \
  -H "ApiKey: YOUR_API_KEY"

And here is the full Python script for a list of domains. The key detail is that with js=false (the default), the Scrape API returns the raw response body as text, so you read resp.text directly rather than parsing JSON:

import requests
import time

API_KEY = "YOUR_API_KEY"
SCRAPE_URL = "https://api.ujeebu.com/scrape"

AI_AGENTS = [
    "gptbot",
    "oai-searchbot",
    "chatgpt-user",
    "claudebot",
    "google-extended",
]

DOMAINS = [
    "scrape.li",
    "example-news-site.com",
    "example-ecommerce.com",
]

def fetch_robots(domain):
    robots_url = f"https://{domain}/robots.txt"
    try:
        resp = requests.get(
            SCRAPE_URL,
            params={
                "url": robots_url,
                "js": "false",
                "proxy_type": "rotating",
                "timeout": 30,
            },
            headers={"ApiKey": API_KEY},
            timeout=40,
        )
        resp.raise_for_status()
        return resp.text  # raw robots.txt content
    except Exception as e:
        print(f"[{domain}] fetch error: {e}")
        return None

def parse_robots(text):
    """
    Parse robots.txt and return a dict of
    {user_agent: {"disallow": [...], "allow": [...]}}
    for the agents we care about, plus the wildcard (*).
    Blank lines and comments are treated as separators
    that end the current user-agent block.
    """
    rules = {}
    current_agents = []

    for raw_line in text.splitlines():
        line = raw_line.strip()

        # Blank lines and comments end the current agent block
        if not line or line.startswith("#"):
            current_agents = []
            continue

        if ":" not in line:
            continue  # malformed line, skip

        field, _, value = line.partition(":")
        field = field.strip().lower()
        value = value.strip()

        if field == "user-agent":
            agent = value.lower()
            current_agents.append(agent)
            if agent not in rules:
                rules[agent] = {"disallow": [], "allow": []}

        elif field == "disallow":
            for agent in current_agents:
                if agent not in rules:
                    rules[agent] = {"disallow": [], "allow": []}
                if value:
                    rules[agent]["disallow"].append(value)

        elif field == "allow":
            for agent in current_agents:
                if agent not in rules:
                    rules[agent] = {"disallow": [], "allow": []}
                if value:
                    rules[agent]["allow"].append(value)

    return rules

def summarise(domain, rules):
    summary = {"domain": domain}
    for agent in AI_AGENTS + ["*"]:
        entry = rules.get(agent, {})
        disallows = entry.get("disallow", [])
        allows = entry.get("allow", [])
        if "/" in disallows:
            status = "full_block"
        elif disallows:
            status = "partial_block"
        elif allows:
            status = "explicit_allow"
        else:
            status = "no_rule"
        summary[agent] = status
    return summary

for domain in DOMAINS:
    body = fetch_robots(domain)

    if body is None:
        print({"domain": domain, "error": "fetch_failed"})
        continue

    # A very short response is likely a 404 or redirect, not a real file
    if len(body.strip()) < 10:
        print({"domain": domain, "error": "empty_or_missing"})
        continue

    rules = parse_robots(body)
    print(summarise(domain, rules))
    time.sleep(0.5)  # polite pacing between requests

The parser walks each line, tracks which User-agent block it is inside, and collects Disallow and Allow rules per agent. Blank lines and comments are treated as separators that end the current user-agent block, which works well in practice for the vast majority of real robots.txt files. Malformed lines (missing colon, unexpected capitalisation) are skipped with a continue rather than crashing the loop. A very short response body is a practical heuristic for a missing or redirected file, not a guaranteed protocol signal.

Takeaway: Set js=false and proxy_type=rotating for robots.txt fetches. Plain text needs no rendering, which keeps the cost at 1 credit per domain and makes the audit affordable at any scale.

Does AI recommend your brand?

Free check: how often ChatGPT, Gemini and Google AI Overviews name your brand next to your top competitor, and whether AI crawlers can read your site. Instant for brands in the Index.

Free. No sign-up. Privacy

Reading the Results: What Block Patterns Mean for Sites That Block AI Crawlers

The parser above produces one of four statuses per agent per domain. Here is what each means and how it affects AI visibility.

Block pattern Affected agents AI-visibility implication
Wildcard full block (User-agent: * / Disallow: /) All crawlers not named above the wildcard Training and retrieval crawlers excluded unless an explicit Allow overrides them
Named full block (User-agent: GPTBot / Disallow: /) Only the named agent Other AI crawlers unaffected; OpenAI training excluded but retrieval agents may still fetch
Named partial block (Disallow: /blog/) Only named agent, only that path Partial content excluded; AI answers may still cite other paths
Google-Extended block Content-use preference only, not a fetch block Google may still index for search; AI training use restricted
Open access (no rule or Allow: /) All crawlers Maximum AI citation potential from both training and retrieval

A wildcard full block is the broadest pattern. Sites that use User-agent: * / Disallow: / and then carve out exceptions with named Allow rules for Googlebot are effectively blocking everything by default. Whether AI crawlers are caught depends on whether they appear in the named exceptions. Always check for explicit Allow rules above the wildcard before concluding that a site is fully open to AI crawlers.

Google-Extended deserves a separate column in your audit output. A site can allow Googlebot to fetch and index every page while disallowing Google-Extended, meaning Google keeps the page in its search index but the content-use preference asks Google not to use it for AI training. This is a deliberate trade-off: search visibility preserved, AI training contribution withheld.

Sites block AI crawlers for several reasons: opposition to unpaid use of their content, concern about aggressive crawling behaviour consuming server resources, or a preference for paid licensing arrangements. There have been documented cases of AI crawlers generating traffic intense enough to affect server performance, which has pushed some site owners toward blocking as a protective measure rather than a policy statement. Sites that leave crawlers open generally want accurate information to appear in AI-generated answers, or simply want to maximise their citation frequency. Both are legitimate positions, and your audit captures which choice each domain has made.

Takeaway: When you see only a wildcard block with no named AI-crawler entries, check for explicit Allow rules above it. That pattern often means the site blocked everything and then carved out exceptions, which may or may not include AI crawlers.

How AI Crawler Access Affects Brand Visibility in AI Answers

The connection between robots.txt and AI citations runs through two separate mechanisms, and conflating them leads to wrong conclusions.

Training-time exclusion happens when a training crawler like GPTBot was blocked during the period when a model was being built. If the crawler respected the Disallow rule, the domain's content is underrepresented in the model's weights. The model is less likely to recall that brand's claims, products or expertise when generating an answer from memory.

Retrieval-time exclusion happens when a retrieval crawler like OAI-SearchBot is blocked today. When ChatGPT uses live web retrieval to answer a question, it cannot fetch pages from domains that block that agent. The domain is less likely to appear as a live citation in answers that use web search, regardless of what the model learned during training.

A brand can be mentioned in an AI answer without being cited as a source, and cited as a source without being recommended. Both matter and need different fixes. For the full framework on how to measure and interpret these distinctions, see our post on what AI visibility means.

After auditing a domain's robots.txt, the natural next step is to check the downstream effect. Ujeebu's AI visibility check shows how often ChatGPT, Gemini and Google AI Overviews name and cite a brand when people ask the questions that brand's buyers ask. If a competitor has an open robots.txt and a high citation frequency, that correlation is a data point for your own access policy review.

The chart below shows which domains appear most often as cited sources in AI answers to buyer questions, drawn from Ujeebu's cited-sources dataset. The distribution is a useful reference when interpreting your audit results: a domain's citation frequency reflects not just its robots.txt policy but also its content type, authority and platform relationships.

YouTube is the most cited website in AI answers to buyer questions, with 4.3% of 12,652 citations, almost all of them from Google AI Overviews.
<title>YouTube is the most cited website in AI answers to buyer questions, with 4.3% of 12,652 citations, almost all of them from Google AI Overviews.</title>youtube.com 543, reddit.com 372, cruisecritic.com 300, facebook.com 202, zendesk.com 193, upstart.com 176, experian.com 174, bankrate.com 168, consumerfinance.gov 168, royalcaribbean.com 165youtube.com<title>youtube.com: 543</title>543reddit.com<title>reddit.com: 372</title>372cruisecritic.com<title>cruisecritic.com: 300</title>300facebook.com<title>facebook.com: 202</title>202zendesk.com<title>zendesk.com: 193</title>193upstart.com<title>upstart.com: 176</title>176experian.com<title>experian.com: 174</title>174bankrate.com<title>bankrate.com: 168</title>168consumerfinance.gov<title>consumerfinance.gov: 168</title>168royalcaribbean.com<title>royalcaribbean.com: 165</title>165

Source: Ujeebu data: Which websites AI answers cite: the 25 most cited domains across four industries

Data
Which websites AI answers cite: the 25 most cited domains across four industries
youtube.com543
reddit.com372
cruisecritic.com300
facebook.com202
zendesk.com193
upstart.com176
experian.com174
bankrate.com168
consumerfinance.gov168
royalcaribbean.com165

YouTube's dominance in that chart, almost entirely through Google AI Overviews, illustrates that platform relationships and content type also shape citation frequency alongside robots.txt policy. robots.txt access is one structural factor, not the only one. For the full cited-sources dataset, see Ujeebu's AI answers cited sources data.

Takeaway: Run the AI visibility check on a competitor after auditing their robots.txt. If they block training crawlers but keep retrieval crawlers open, they are making a deliberate trade-off. If they block everything, their lower citation frequency is partly a policy choice.

What the Data Shows: Early Findings from the Top 1,000 Sites

Ujeebu is publishing a dataset tracking GPTBot, ClaudeBot and Google-Extended access across the top 1,000 sites, with the first release scheduled for December 2026. You can find it at /data when it publishes.

In the meantime, the cited-sources data already points to a pattern worth noting: the domains that appear most frequently as AI citations tend to be platforms with open access policies, high content volume and strong domain authority. Domains that have blocked training crawlers can still appear in retrieval-based citations if they allow retrieval agents, but their representation in model weights is lower, which affects the quality and frequency of unprompted mentions.

The audit script above is the tool to run against your own list before the dataset publishes. The results will tell you which competitors have already made a policy choice and which have not yet updated their robots.txt to address AI crawlers at all.

Keeping Your Audit Fresh: Scheduling Recurring robots.txt Checks

A one-off audit is a photograph. A recurring check is a time-lapse, and the time-lapse is what catches the moment a competitor quietly opens or closes the door to AI crawlers.

robots.txt policies can change as site owners adjust to new options or experiences. A site that allowed GPTBot last month may have added a Disallow rule this month after a bad crawling experience. Conversely, a site that blocked all AI crawlers may have opened access after signing a licensing arrangement. Either change is a signal worth acting on.

The simplest monitoring setup stores the parsed summary dict (domain, agent, status, timestamp) in a database or flat JSON file after each run, then diffs the new result against the stored one. If any agent's status changes, send a notification. This is the same pattern used for SEO monitoring: fetch, parse, compare, alert.

At 1 credit per domain, running the audit weekly across 500 domains costs 500 credits per week, well within any paid plan. For a breakdown of per-request costs across plans, see the pricing page.

Takeaway: Run the audit on a weekly schedule and store diffs. A competitor adding a GPTBot Disallow rule is a meaningful competitive signal: it may precede a licensing negotiation or a content-protection push that affects their AI visibility relative to yours.


The free trial includes 5,000 credits with no credit card required, enough to audit up to 5,000 domains. Sign up and run the script against your own list.


FAQ

What is GPTBot and how do I block it in robots.txt?

GPTBot is OpenAI's training crawler, identified by the user-agent string GPTBot. To block it, add User-agent: GPTBot followed by Disallow: / to your robots.txt file. OpenAI documents GPTBot and states that it respects these rules. Note that blocking GPTBot does not block OAI-SearchBot or ChatGPT-User, which are separate OpenAI agents that each require their own Disallow entry.

Does blocking GPTBot also block ChatGPT from citing my site?

Not automatically. GPTBot is a training crawler; blocking it affects what OpenAI's models learned during training. ChatGPT's live web retrieval uses a separate agent, OAI-SearchBot. If you block GPTBot but not OAI-SearchBot, ChatGPT can still fetch your pages when answering questions with web search enabled. To reduce citation frequency across both training and retrieval, you need separate Disallow rules for each agent.

Which AI crawlers respect robots.txt and which ignore it?

Anthropic documents that ClaudeBot respects robots.txt Disallow rules. OpenAI states that GPTBot respects robots.txt. robots.txt is a convention with no technical enforcement mechanism, so compliance depends on each operator's good faith. An audit of robots.txt tells you what a site's declared policy is, not whether every crawler that visits actually obeys it.

How do I check robots.txt for hundreds of domains without getting rate-limited?

Route each request through a rotating proxy pool so the traffic does not accumulate on a single IP. The Ujeebu Scrape API handles this with proxy_type=rotating. Because robots.txt is plain text, set js=false to keep the cost at 1 credit per domain. The Python script in this post loops over a domain list, fetches each robots.txt via the API, and parses the result, with a short sleep between requests as an additional courtesy.

Does blocking AI crawlers hurt my Google search rankings?

Blocking AI crawlers does not directly affect Google search rankings, because Googlebot is a separate agent from Google-Extended. A site can disallow Google-Extended (which controls AI training and Gemini content use) while leaving Googlebot fully permitted, so search indexing continues normally. The risk is to AI visibility in generative answers, not to traditional search rankings.

How often do sites change their AI crawler rules?

Policies can change as site owners adjust to new options or experiences with specific crawlers. A recurring weekly check with a diff against stored results is the most reliable way to detect changes. A competitor adding or removing an AI crawler block is a meaningful competitive signal worth tracking.

Try it yourself

Run this on your own pages

Get a free API key and make your first call in minutes. 5,000 free credits for 7 days, no card.

← Back to blog Read the docs →