robot_2Free · No sign-up

Can AI crawlers read your site?

Enter your domain and we’ll read your robots.txt to see whether ChatGPT, Gemini, Claude, Perplexity and the search engines are allowed in. If you’re accidentally blocking them, we’ll show you the exact rule that’s doing it.

lockWe only read the public robots.txt file at your domain. Nothing is stored.

robots.txt is a small text file at the root of your domain that tells automated crawlers which parts of your site they may visit. Search engines have used it for decades, and now the AI companies have their own crawlers that respect it too: OpenAI’s GPTBot, Google’sGoogle-Extended, Anthropic’s ClaudeBot,PerplexityBot and others. If your robots.txt disallows one of them, that assistant simply can’t read your pages.

The problem is that plenty of small business sites block these crawlers without meaning to, usually from an old rule, a cautious template, or an SEO plugin’s default. The result is that a business doing everything else right is invisible to the very assistants their customers are starting to ask. This tool reads your live robots.txt, works out how each crawler would interpret it for your homepage, and shows you in plain English who is allowed in and who is blocked, including the exact rule responsible.

Worth knowing: if you have no robots.txt at all, or it doesn’t mention a crawler, that crawler is allowed by default, so no news is good news here. For the fuller picture of whether assistants can understand and recommend you, not just reach you, pair this with the AI Visibility Checker.

AI Visibility Checkerarrow_forwardHow AI sees your businessarrow_forwardAI optimisation (GEO) servicearrow_forward

FAQ

Common questions

Why would I be blocking AI crawlers by accident?
Lots of sites added rules to block bots years ago, or inherited a robots.txt from a template or an SEO plugin, without realising it now blocks AI crawlers like GPTBot or Google-Extended. If those are blocked, assistants like ChatGPT and Gemini cannot read your site, so they cannot learn to recommend you.
Should I allow or block AI crawlers?
For most small businesses that want to be found and recommended, you should allow them: being readable by ChatGPT, Gemini, Claude and Perplexity is quickly becoming as important as being in Google. Some publishers block AI crawlers to protect paid content, but if your goal is visibility, blocking them works against you.
What is the difference between GPTBot, OAI-SearchBot and ChatGPT-User?
GPTBot crawls pages to train and improve OpenAI’s models, OAI-SearchBot powers results shown inside ChatGPT search, and ChatGPT-User fetches a specific page when someone clicks or shares a link in ChatGPT. Blocking one does not block the others, so it is worth checking all three.
How do I unblock an AI crawler?
Edit the robots.txt file at the root of your domain and remove or narrow the Disallow rule under that crawler’s User-agent. If you are not comfortable editing it, your web person or platform support can do it in minutes, and this tool shows you exactly which rule is blocking each bot.
Does a missing robots.txt mean everything is blocked?
No, the opposite. If there is no robots.txt, or it does not mention a crawler, that crawler is allowed by default. robots.txt only restricts what it explicitly disallows.
chat