Free tool

Free AI Crawler Checker

See whether your robots.txt allows or blocks GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and other AI crawlers for one path.

robots.txt

Fetches /robots.txt from that host. The address is not stored.

What the checker is reading

robots.txt sits at the root of a host, for example https://example.com/robots.txt. This page fetches that file and tests one path. The default is /, the home page. A Disallow for /private says nothing about the home page, so check the path you care about.

Matching follows RFC 9309. Crawler names ignore case. The longest matching Allow or Disallow wins, and Allow wins when the two patterns are the same length. A star in a pattern means any run of characters. A dollar sign at the end means the path must end there. If the crawler has no group of its own, the star group is used. If that group is missing too, nothing is disallowed.

A 404 is not a block. RFC 9309 treats an unavailable file, including other 4xx responses except 429, as permission to fetch. A 5xx response or a failed connection is the opposite: the spec says crawlers must assume complete disallow. Google describes using a cached copy and, with none, assuming no restrictions. Amazon says its crawlers treat a file they cannot fetch as if it does not exist. The result on this page says which case the check hit.

Training, search, and a fetch someone asked for

Training tokens are the ones a vendor says may feed a model: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Amazonbot, MistralAI-Training, and CCBot. CCBot builds Common Crawl's public dataset, and the opt-out token is CCBot. Google-Extended is not a separate downloader. Google says it controls Gemini training and grounding, and that it does not affect Google Search inclusion or ranking. Applebot-Extended does not crawl on its own. Blocking it leaves Spotlight, Siri, and Safari search in place.

Search and answer tokens build an index a product can cite. OAI-SearchBot, Claude-SearchBot, PerplexityBot, DuckAssistBot, Amzn-SearchBot, and MistralAI-Index are in that group, with Googlebot and Bingbot as the ordinary web-search baseline. The settings are independent. Disallowing GPTBot does not disallow OAI-SearchBot, and disallowing Google-Extended does not disallow Googlebot. OpenAI says a site that blocks OAI-SearchBot is left out of ChatGPT search answers. Perplexity says PerplexityBot is how pages get into its search, and that the crawler is not used for training. Blocking a search or answer crawler can remove you from that assistant's answers. It does not block the others, and it is not a Google ranking change.

User-triggered fetchers run because a person asked: ChatGPT-User, Claude-User, Perplexity-User, Amzn-User, and MistralAI-User. OpenAI says robots.txt may not apply to ChatGPT-User. Perplexity says Perplexity-User generally ignores the file. Amazon says Amzn-User may skip directives. A Disallow there is a request, not a lock. Anthropic says a Disallow does stop Claude-User. The result column is what the file says. Apple also says that if a file names Googlebot and not Applebot, Applebot follows the Googlebot group. This checker does that, and it does not copy that fallback onto other vendors.

Allow search crawlers and block training

Use this as a new file, or merge it into the one you already serve. A named group is not overridden by the star group. If the same token already has rules, the two groups combine and the longest match still wins. ChatGPT-User, Perplexity-User, and Amzn-User are left unnamed on purpose, because their vendors say a robots.txt line may not control those fetches. They fall through to the star group, which allows the site. OAI-AdsBot is unnamed too. OpenAI says it only visits pages submitted as ads.

# Training tokens. A named group is not overridden by the * group below.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Amazonbot
User-agent: MistralAI-Training
User-agent: CCBot
Disallow: /

# Search and answer tokens, including Googlebot and Bingbot.
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Googlebot
User-agent: Bingbot
User-agent: Applebot
User-agent: Amzn-SearchBot
User-agent: MistralAI-Index
User-agent: DuckAssistBot
Allow: /

# Crawlers not named above. This does not cancel a named Disallow.
User-agent: *
Allow: /

Allow every crawler

A file that allows everything can be this short. A named Disallow in the same file still applies, because a named group beats the star group. To open a crawler you had blocked, delete that group or change the rule. Allow: / does not cancel a longer Disallow on the same path. Publishing no robots.txt, or serving a 404, also leaves paths allowed. That is easy to confuse with a host that is down. If the file cannot be reached, do not assume the site is open.

User-agent: *
Allow: /

llms.txt is not robots.txt

llms.txt is a plain-text index some sites publish so a model can find pages. Posting Dude publishes one at /llms.txt. It describes the site. It does not allow or block a crawler. robots.txt is the access file. A URL in llms.txt does not override a Disallow, and a Disallow does not edit llms.txt. If you want a bot to read a page, the robots.txt rule and the page both have to permit it. Check robots.txt when the question is whether GPTBot, ClaudeBot, or PerplexityBot may fetch a path. Check llms.txt when the question is whether you published a map.

Also check your llms.txt.

Check the live file after you deploy

A deploy can replace robots.txt, and a CDN can keep the previous copy. Check the live hostname after the release. www, the apex host, and each subdomain have their own file. OpenAI and Perplexity say an edit can take about a day to show up in search. DuckDuckGo says 72 hours for DuckAssistBot. This checker only reads the file that is live now. Allowed means a complying crawler may fetch that path. Blocked means the last column is the longest matching line. Neither result is a ranking.

The meta title checker previews a title and description before you publish. The UTM link builder tags a campaign URL. The free tools index lists the other tools on this site. Pricing lists Launch and Studio. The why page covers what happens when distribution stops after launch day. The monthly distribution mix explains the billing-month package.

Related: llms.txt checker, Schema markup checker, and Schema markup generator. The free tools index has the rest.

Questions about AI crawlers and robots.txt

What is GPTBot?+

GPTBot is OpenAI's crawler for pages that may be used to train generative models. The robots.txt token is GPTBot. It is separate from OAI-SearchBot, which OpenAI uses for ChatGPT search, and from ChatGPT-User, which fetches a page when a person asks ChatGPT to look at it. OpenAI says those settings are independent.

Does blocking GPTBot hurt my Google rankings?+

No. GPTBot is OpenAI's token. Google Search uses Googlebot. Google-Extended is Google's control for Gemini training and grounding, and Google says it does not affect inclusion or ranking in Google Search. Blocking GPTBot does not change the Googlebot rule. Blocking Googlebot can keep a page out of Google Search.

What's the difference between training and search crawlers?+

A training crawler collects pages that may be used to train a model. A search or answer crawler builds an index so the product can link or cite the page. The token names are not interchangeable. Disallowing GPTBot does not disallow OAI-SearchBot, and disallowing Google-Extended does not disallow Googlebot. Blocking a search crawler can remove you from that product's answers.

How do I allow AI crawlers in robots.txt?+

Put the file at the root of the host. Give each crawler you want to allow its own group with Allow: /, or leave it to a star group that allows the path. A named Disallow beats the star group. User-agent matching ignores case. After you deploy, fetch the live file again. A 404 means there is no file, which allows crawling. If the file is unreachable or the server returns 5xx, RFC 9309 says crawlers must assume complete disallow until they can read rules.

Does robots.txt stop all scraping?+

No. robots.txt is a voluntary standard. Crawlers that choose to comply will follow it. It is not authentication, and it does not stop a client that ignores the file. Listing a path in the file also publishes that path. Use access controls on the server for anything that must stay private. Some user-triggered fetchers, including ChatGPT-User and Perplexity-User, may ignore the file even when that vendor's other bots honor it.

Done-for-you distribution

Want the pages written and posted for you?

Posting Dude publishes human-written distribution every month. Launch is $249/mo for one product. Studio is $399/mo for up to three products. Both plans publish the same ten editorial articles per project. See live examples of published articles, then compare plans on pricing.