← All tools

Seo checker

AI Crawler Checker

Enter your website. SEOGrove reads your robots.txt and shows which AI crawlers can visit your homepage and which are blocked.

Runs instantly. No account required.

How to use the AI crawler checker

  1. Enter your website address, like example.com.
  2. Run the check. SEOGrove fetches /robots.txt from your site.
  3. Read the list. Each crawler shows allowed or blocked, plus who runs it and what it is for.

If your site has no robots.txt, every crawler is allowed.

The crawlers we check

Crawler Company What it does
GPTBot OpenAI Collects pages to train models
OAI-SearchBot OpenAI Finds pages for ChatGPT search
ChatGPT-User OpenAI Opens pages a ChatGPT user asks about
ClaudeBot Anthropic Collects pages to train models
Claude-SearchBot Anthropic Finds pages for Claude’s search
PerplexityBot Perplexity Finds pages for Perplexity answers
Google-Extended Google Controls use in Gemini (not Google Search)
Applebot-Extended Apple Controls use in Apple Intelligence
CCBot Common Crawl Builds an open dataset many AI models use
meta-externalagent Meta Collects pages for Meta AI

Should you allow AI crawlers?

It depends on what you want.

  • If you want to be mentioned in AI answers, allow the search crawlers at least: OAI-SearchBot, Claude-SearchBot and PerplexityBot. If they can’t read your pages, they can’t cite them.
  • If you don’t want your content used for training, you can block the training crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended) and still allow the search ones.

Many sites block AI crawlers by accident. A security plugin, a CDN setting or an old “block all bots” rule can do it without anyone noticing. Run this check after any big site change.

Example: block training, allow AI search

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
Disallow: /

User-agent: *
Allow: /

Frequently asked questions

What does the AI crawler checker do?

It reads your site's robots.txt and works out, for 10 well-known AI crawlers, whether each one may visit your homepage. It follows the official robots.txt rules (RFC 9309): the most specific user-agent group applies, and the longest matching rule wins.

If I block GPTBot, will I disappear from ChatGPT?

Not exactly. OpenAI uses GPTBot to collect training data, OAI-SearchBot to find pages for ChatGPT search, and ChatGPT-User when a user asks ChatGPT to open a page. Blocking OAI-SearchBot is what can keep your pages out of ChatGPT search answers.

Does blocking Google-Extended remove me from Google Search?

No. Google-Extended only controls whether Google may use your content for Gemini models and grounding. It does not change your Google Search rankings. AI Overviews in Google Search are based on Googlebot, not Google-Extended.

Do all AI crawlers obey robots.txt?

The major ones say they do. robots.txt is a request, not a lock, so a badly behaved crawler can ignore it. To truly block a bot, you need server or firewall rules.

Why does it only check the homepage?

The homepage is the quickest test of whether a crawler is welcome at all. If you block only some folders, like /blog, a crawler can still be allowed on the homepage. Read your robots.txt to check specific paths.

Why did it say it couldn't read my robots.txt?

Your server didn't respond or returned an error. Under the robots.txt standard, crawlers treat a server error as "stay out" until it is fixed, so this is worth fixing quickly.