Is your website blocking ChatGPT?
Many sites block AI assistants without meaning to, often through a security plugin or a blanket rule. Enter your website to see which AI crawlers your robots.txt lets in, and what each block costs you.
Search crawlers matter more than training crawlers
AI companies run different crawlers for different jobs. Search crawlers (such as OAI-SearchBot, Claude-SearchBot and PerplexityBot) find the pages an assistant shows and links in its answers. Block them and your pages cannot be quoted. Training crawlers (such as GPTBot, ClaudeBot and CCBot) collect pages to train future models. Blocking those keeps your content out of training, which some sites want, and it does not by itself remove you from today's answers.
User-requested fetchers (ChatGPT-User, Claude-User, Perplexity-User) open a page when a person asks about it in a chat. The companies treat robots.txt differently for these, so a block may not stop every visit.
How to let the search crawlers in
If you want to be recommended, make sure your robots.txt does not disallow the search crawlers. A file that allows everyone except a private folder looks like this:
User-agent: * Allow: / Disallow: /admin/ Sitemap: https://yourbusiness.com/sitemap.xml
To keep training crawlers out but stay in search answers, add groups for the training bots only, for example User-agent: GPTBot followed by Disallow: /. Leave OAI-SearchBot, Claude-SearchBot and PerplexityBot alone.
Questions
Does allowing the crawlers mean ChatGPT will recommend me?
No. It only means the assistant can read your pages. Whether it names you depends on what the pages it trusts say about your market. The free visibility check shows which pages those are.
Where does this tool look?
It reads your public robots.txt and checks whether you have an llms.txt file. It does not crawl your pages. Requests come from BoldlistedCheck, described on our crawler page.
My robots.txt looks fine but I am still not named. Why?
Some firewalls and security plugins block AI crawlers before robots.txt is ever read, and some sites are simply not mentioned on the pages assistants rely on. Run the free check to see which it is.
See what ChatGPT says about your business
The free check asks ChatGPT and Gemini ten questions your buyers ask, as a real person in your country sees them, and shows who they name instead of you. About two minutes, no sign-up.
Run the free check