AI Search Visibility Checker

Last updated 23 September 2026. Bot names and purposes checked against each company's own documentation.

Why AI crawler access matters

When someone asks ChatGPT, Perplexity, Claude, Copilot or Google's AI Mode to recommend a business, the answer is built from pages those tools can reach. If your website blocks their bots, you can't be cited, however good your business is.

This happens more often than you'd think. Many websites block AI bots by accident. It might be a copied robots.txt template, a security plugin, or a hosting or CDN setting nobody remembers switching on. The checker above shows in seconds whether that's happening to you, and writes the fix.

The three kinds of AI bots

Most AI companies now run separate bots for different jobs, and you can treat each one differently:

The companies say these controls are independent. OpenAI, for example, says a site can allow OAI-SearchBot to appear in ChatGPT search while disallowing GPTBot. Blocking a training bot doesn't remove you from that company's AI search.

Should you block AI training?

There's no single right answer:

• Allow training if you want AI models to know your business. When models learn about you during training, they can mention you even without searching the web. For most small businesses that want to be recommended, this is the better choice.
• Block training if your content is your product. Publishers, writers, photographers and course creators may not want their original work used to train models. You can still stay visible in AI search.

A few bots behave differently

• Google-Extended isn't a separate crawler. It's a token that controls whether Gemini can train on your content and use it for grounding in Gemini apps. Google says it doesn't affect Google Search or rankings, and AI Overviews and AI Mode are part of Search. To stay in those, you need Googlebot allowed.
• Applebot-Extended doesn't crawl either. Blocking it stops Apple training on your content, but Apple says you still appear in Siri, Spotlight and Safari.
• Some user-request fetchers can ignore robots.txt. Perplexity says Perplexity-User generally ignores it, Meta says its fetcher may bypass it, and OpenAI says robots.txt rules may not apply to ChatGPT-User because a person asked for the page.
• Microsoft Copilot draws on Bing's index, so blocking Bingbot hurts your chances there too.

How to check and fix your robots.txt

1. Open yoursite.co.uk/robots.txt in your browser and copy everything.
2. Paste it into the checker above and click "Check my site".
3. If anything important is blocked, choose "Maximum AI visibility" or "Visible in AI search, no training". The tool rewrites your file, keeps your other rules and sitemap, and re-tests the result.
4. Copy the new file into your site. In WordPress with Yoast SEO, go to Yoast SEO → Tools → File editor. With Rank Math, go to General Settings → Edit robots.txt.
5. Reload your robots.txt and check again.

Here's what a simple file that allows AI search but blocks training looks like:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
Disallow: /

Sitemap: https://yoursite.co.uk/sitemap_index.xml

A bot follows the group with its own name if there is one. Otherwise it follows the User-agent: * group. The checker uses Google's published matching rules, including wildcards and "longest rule wins".

Common robots.txt mistakes that hide you from AI

• A leftover "Disallow: /" from development. Sites built on a staging server sometimes go live still blocking everything. That hides you from Google and Bing too. The checker flags it as critical.
• A copied "block all AI" template. Some popular templates block OAI-SearchBot and PerplexityBot along with the training bots, which removes you from AI search as well as training.
• Blocking ChatGPT-User to stop training. ChatGPT-User only opens pages when a person asks, so blocking it doesn't stop training. It just stops ChatGPT reading your page for that person. To opt out of training, block GPTBot instead.
• Assuming Google-Extended controls AI Overviews. It doesn't. Blocking it doesn't remove you from AI Overviews, and allowing it doesn't get you in.
• Using "Noindex" in robots.txt. Google doesn't support it there. Use a noindex meta tag on the page instead.
• No sitemap line. Adding Sitemap: with your sitemap address helps every crawler find all your pages.

Don't forget your firewall or CDN

robots.txt is only a request. Services like Cloudflare can block AI bots at the network level, before they ever read your robots.txt. Cloudflare lets you set AI bot policies in your security settings (also called AI Crawl Control). Since 15 September 2026, new domains on Cloudflare block AI training and agent bots on pages that show ads by default, while search bots stay allowed. That matters if you run ads, for example Google AdSense. If you use Cloudflare, check those settings match what you want. If the robots.txt you paste includes a "Cloudflare Managed" section, the checker spots it and tells you to change it in Cloudflare.

The truth about llms.txt

llms.txt is a proposed standard: a plain-text summary of your site for AI tools. You'll find plenty of pages claiming it's essential. Google's own guidance on optimising for its AI features says otherwise:

"You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search."

Google says such files neither help nor harm your visibility in Search. Some AI companies publish llms.txt files for their own developer documentation, but none has confirmed that having one makes AI tools cite you more. Our generator is there if you want one, as it takes two minutes and does no harm. Just don't expect it to move the needle. Crawler access matters far more.

Does schema help with AI search?

Google is clear here too: structured data isn't required for its generative AI features, and there's no special schema for them. It still recommends structured data as part of good SEO. For a local business, accurate markup of your name, address, phone number, opening hours and profiles gives search engines a consistent description of you. That's why the business schema generator is in this tool. It's a solid foundation, not a shortcut.

If you've added FAQ schema to get the question-and-answer dropdowns in Google results, note that Google stopped showing FAQ rich results on 7 May 2026.

How to read your AI answer test

The AI answer test writes the questions real customers ask, then lets you open each one in ChatGPT, Perplexity or Google AI Mode. Record whether you were mentioned, and whether the details were right.

• Mentioned in discovery questions ("Who is the best… in Bristol?") is the strongest sign. AI tools see you as a relevant choice.
• Known by name, but missing from discovery answers usually means AI tools know you exist but don't have enough independent sources to recommend you. Reviews, directory listings and mentions on other sites help most.
• Wrong details, like an old address, phone number or opening hours, usually come from out-of-date listings. Fix your Google Business Profile and main directories first.
• Not known at all often means an access problem (check tab 1) or a site that doesn't clearly say what you do and where.

Answers vary between tools, sessions and days, so treat one check as a snapshot. Use a private browser window, ask each question more than once, and repeat the test every month or two to see the trend.

What actually improves AI visibility

AI tools recommend businesses they can find, understand and trust. Based on what the AI companies and Google have published, the reliable steps are:

1. Be reachable. Allow AI search bots in robots.txt and in your firewall or CDN.
2. Be indexed. Check Google Search Console, including the generative AI performance report added in June 2026. Also verify your site in Bing Webmaster Tools, as Copilot draws on Bing.
3. Be clear. Have pages that plainly say what you do, where and for whom, with prices or starting prices if you can. Answer the questions customers actually ask.
4. Be consistent. Keep your name, address and phone number identical on your site, Google Business Profile and directories. AI tools often repeat what those sources say.
5. Be mentioned. Reviews, local press, industry directories and partner websites give AI tools independent sources to cite.
6. Check regularly. Use the AI answer test above every month or two. AI answers change as the tools update.

Frequently asked questions

How do I know if ChatGPT can access my website?

Check your robots.txt for rules about OAI-SearchBot, which powers ChatGPT search, and ChatGPT-User, which opens pages when users ask. Paste your robots.txt into the checker above to see instantly whether they're allowed.

Does blocking GPTBot remove me from ChatGPT search?

No. GPTBot collects content for training. ChatGPT search uses OAI-SearchBot, and OpenAI says each setting is independent.

Does blocking Google-Extended remove me from AI Overviews?

No. Google says Google-Extended doesn't affect inclusion in Google Search. AI Overviews and AI Mode are part of Search and rely on Googlebot.

Do I need an llms.txt file?

No. Google says you don't need AI text files to appear in its search features. An llms.txt file won't hurt, but no major AI company has confirmed it increases citations.

Do all AI bots obey robots.txt?

The main crawlers say they do. Some user-request fetchers, such as Perplexity-User, say they may ignore robots.txt because a person asked for the page.

Should I block AI bots from training on my website?

It depends on your goals. Most small businesses benefit from allowing training, as it helps AI models learn about them. If your content is what you sell, such as articles, photos or courses, blocking training bots while allowing AI search bots is a sensible middle ground.

Why am I still blocked after fixing robots.txt?

Check your firewall or CDN. Cloudflare and some hosts can block AI bots before they read robots.txt. Also allow time: some bots only re-read robots.txt every day or so.

More free tools for UK businesses

• Conversion Rate BenchmarksCompare against 20+ industries
• Marketing Budget OptimiserPlan spend across channels
• AI Visibility ServicesGet recommended by AI tools
• All free business toolsSee all our calculators

Sources: OpenAI crawlers; Anthropic crawlers; Perplexity crawlers; Google crawlers; Google: optimising for AI features; Google robots.txt rules; Applebot; Meta crawlers; Amazonbot; Mistral; Cloudflare managed robots.txt; llms.txt proposal.