AI crawler check
See which AI crawlers your robots.txt lets in: the bots behind ChatGPT, Claude, Perplexity, Gemini and Copilot answers, and the ones that collect training data. Free, no account.
What the three kinds of crawler do
AI search and answers. Bots such as OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and bingbot index pages that AI answers then draw on and cite. Shutting one out keeps your pages out of that AI search engine's answers.
Fetches for a user. Bots such as ChatGPT-User, Claude-User and Perplexity-User fetch a page because someone asked an assistant about it just now. Some of them say they ignore robots.txt, as the request is a person's.
Model training. Bots such as GPTBot, ClaudeBot and CCBot, and controls such as Google-Extended, collect content to train models. Whether to allow them is your choice; it does not decide whether answers can cite you.
The check reads robots.txt only, the way the major crawlers read it, and keeps no record of the site or of you. The same check runs in every Queryosity website audit. See the methodology and the bot that asks.