What the checker is reading
robots.txt sits at the root of a host, for example https://example.com/robots.txt. This page fetches that file and tests one path. The default is /, the home page. A Disallow for /private says nothing about the home page, so check the path you care about.
Matching follows RFC 9309. Crawler names ignore case. The longest matching Allow or Disallow wins, and Allow wins when the two patterns are the same length. A star in a pattern means any run of characters. A dollar sign at the end means the path must end there. If the crawler has no group of its own, the star group is used. If that group is missing too, nothing is disallowed.
A 404 is not a block. RFC 9309 treats an unavailable file, including other 4xx responses except 429, as permission to fetch. A 5xx response or a failed connection is the opposite: the spec says crawlers must assume complete disallow. Google describes using a cached copy and, with none, assuming no restrictions. Amazon says its crawlers treat a file they cannot fetch as if it does not exist. The result on this page says which case the check hit.
Training, search, and a fetch someone asked for
Training tokens are the ones a vendor says may feed a model: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Amazonbot, MistralAI-Training, and CCBot. CCBot builds Common Crawl's public dataset, and the opt-out token is CCBot. Google-Extended is not a separate downloader. Google says it controls Gemini training and grounding, and that it does not affect Google Search inclusion or ranking. Applebot-Extended does not crawl on its own. Blocking it leaves Spotlight, Siri, and Safari search in place.
Search and answer tokens build an index a product can cite. OAI-SearchBot, Claude-SearchBot, PerplexityBot, DuckAssistBot, Amzn-SearchBot, and MistralAI-Index are in that group, with Googlebot and Bingbot as the ordinary web-search baseline. The settings are independent. Disallowing GPTBot does not disallow OAI-SearchBot, and disallowing Google-Extended does not disallow Googlebot. OpenAI says a site that blocks OAI-SearchBot is left out of ChatGPT search answers. Perplexity says PerplexityBot is how pages get into its search, and that the crawler is not used for training. Blocking a search or answer crawler can remove you from that assistant's answers. It does not block the others, and it is not a Google ranking change.
User-triggered fetchers run because a person asked: ChatGPT-User, Claude-User, Perplexity-User, Amzn-User, and MistralAI-User. OpenAI says robots.txt may not apply to ChatGPT-User. Perplexity says Perplexity-User generally ignores the file. Amazon says Amzn-User may skip directives. A Disallow there is a request, not a lock. Anthropic says a Disallow does stop Claude-User. The result column is what the file says. Apple also says that if a file names Googlebot and not Applebot, Applebot follows the Googlebot group. This checker does that, and it does not copy that fallback onto other vendors.
Allow search crawlers and block training
Use this as a new file, or merge it into the one you already serve. A named group is not overridden by the star group. If the same token already has rules, the two groups combine and the longest match still wins. ChatGPT-User, Perplexity-User, and Amzn-User are left unnamed on purpose, because their vendors say a robots.txt line may not control those fetches. They fall through to the star group, which allows the site. OAI-AdsBot is unnamed too. OpenAI says it only visits pages submitted as ads.
# Training tokens. A named group is not overridden by the * group below. User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: Amazonbot User-agent: MistralAI-Training User-agent: CCBot Disallow: / # Search and answer tokens, including Googlebot and Bingbot. User-agent: OAI-SearchBot User-agent: Claude-SearchBot User-agent: PerplexityBot User-agent: Googlebot User-agent: Bingbot User-agent: Applebot User-agent: Amzn-SearchBot User-agent: MistralAI-Index User-agent: DuckAssistBot Allow: / # Crawlers not named above. This does not cancel a named Disallow. User-agent: * Allow: /
Allow every crawler
A file that allows everything can be this short. A named Disallow in the same file still applies, because a named group beats the star group. To open a crawler you had blocked, delete that group or change the rule. Allow: / does not cancel a longer Disallow on the same path. Publishing no robots.txt, or serving a 404, also leaves paths allowed. That is easy to confuse with a host that is down. If the file cannot be reached, do not assume the site is open.
User-agent: * Allow: /
llms.txt is not robots.txt
llms.txt is a plain-text index some sites publish so a model can find pages. Posting Dude publishes one at /llms.txt. It describes the site. It does not allow or block a crawler. robots.txt is the access file. A URL in llms.txt does not override a Disallow, and a Disallow does not edit llms.txt. If you want a bot to read a page, the robots.txt rule and the page both have to permit it. Check robots.txt when the question is whether GPTBot, ClaudeBot, or PerplexityBot may fetch a path. Check llms.txt when the question is whether you published a map.
Check the live file after you deploy
A deploy can replace robots.txt, and a CDN can keep the previous copy. Check the live hostname after the release. www, the apex host, and each subdomain have their own file. OpenAI and Perplexity say an edit can take about a day to show up in search. DuckDuckGo says 72 hours for DuckAssistBot. This checker only reads the file that is live now. Allowed means a complying crawler may fetch that path. Blocked means the last column is the longest matching line. Neither result is a ranking.
The meta title checker previews a title and description before you publish. The UTM link builder tags a campaign URL. The free tools index lists the other tools on this site. Pricing lists Launch and Studio. The why page covers what happens when distribution stops after launch day. The monthly distribution mix explains the billing-month package.