AI companies use several different crawlers, and they do different jobs. Some fetch pages so the site can appear in AI search answers, such as OAI-SearchBot for ChatGPT search, Claude-SearchBot and PerplexityBot. Some collect content to train models, such as GPTBot, ClaudeBot and CCBot. Some only fetch a page when a user asks for it. Google-Extended and Applebot-Extended are not separate crawlers at all: they are names you use in robots.txt to opt out of Gemini or Apple model training without affecting normal search.
This checker reads your robots.txt and the page you enter, then shows for each crawler whether it is allowed, the exact robots.txt line that decided it, and any meta robots tag or X-Robots-Tag header that names it. Matching follows the robots.txt standard (RFC 9309): the most specific user-agent group applies, and the longest matching rule wins, with Allow winning a tie. Crawler names and purposes come from each company’s own documentation, checked on 4 November 2026.
Robots.txt is a request, not a lock. Crawlers from major companies say they follow it, but user-triggered fetchers may not, and some crawlers ignore it. A firewall, CDN or security plugin can also block a crawler that robots.txt allows, which this tool cannot see. Blocking search crawlers can keep you out of those answers, so decide crawler by crawler.