Skip to main content
Free tool

AI visibility checker

A free check of one page for AI crawlers, robots.txt for 14 crawlers, text without JavaScript, directives and schema

  • Free
  • No account required

Our server fetches the URL, plus that domain's robots.txt and llms.txt, so it reaches our logs. Private and local addresses are rejected, including after a redirect.

This tool checks AI crawler access and whether a public page is readable. It does not measure citations or rankings in ChatGPT, Perplexity, Gemini or AI Overviews. Readiness checks and observations of selected answers have separate scopes.

This AI visibility checker reads one public page the way an AI crawler gets it: the server HTML, robots.txt and the indexing directives, skipping JavaScript. It checks whether AI search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot may fetch the page and how much text is left without JavaScript. It also checks whether noindex or nosnippet limits quoting, and which structured data and llms.txt the site has. The result shows readiness to be read. This tool does not collect AI-system answers or measure citations. Mentions and links in a selected sample of answers need a separate observation, with the question, system and date recorded.

About the tool

What to fix first

Start with robots.txt. A single Disallow: / under the wrong user agent hides the page from an AI search engine. Allow the search crawlers you care about, and decide about training crawlers separately. You can test each rule in our robots.txt tester.

Then check the text without JavaScript. Open the page source (Ctrl+U) and search for a sentence from your main content. If it is missing there, a crawler that skips scripts misses it too. Server-side rendering or prerendering fixes that for the whole site.

Directives and schema come next. A noindex or nosnippet left over from a staging setup is a common reason a page is missing from results. Structured data and an llms.txt file help machines read the page, with no promise of citations.

This is one page at one moment. To check every page and repeat the check after each release, see AI search readiness in CometWeb Insight.

In practice

Check several kinds of page

robots.txt rules and rendering often differ between templates. Check the home page, one article or product and a category page. If only one template has little text in the server HTML, the fix belongs to that template, not to the whole site.

Google Search Central, AI features and your website
Step by step

How to check if AI crawlers can read your page

You check a page for AI crawlers in three places: robots.txt, the HTML the server sends and the robots directives. Five steps show the order to work in.

  1. Run the check on one page

    Enter the URL of a page you care about, such as a product page. The tool fetches it, robots.txt and /llms.txt, and the result shows the status for each crawler separately.

  2. Read the robots.txt rules for each token

    A crawler looks for a group with its own name, ignoring case, and otherwise falls back to User-agent: *. One Disallow: / in the general group therefore covers every bot without its own group.

  3. Check the text in the server HTML

    Open the page source (Ctrl+U) and search for a sentence from the main content. If it is missing there and appears only after JavaScript runs, a crawler that skips scripts sees an empty page.

  4. Check for noindex and nosnippet

    Look in <meta name="robots"> and in the X-Robots-Tag header. In Google, nosnippet also stops the content being used as direct input for AI Overviews and AI Mode.

  5. Decide on search and training separately

    OpenAI, Anthropic and Google keep search crawlers apart from the ones that collect training data. Write down a decision for each group and set your robots.txt rules to match it.

Examples

Example robots.txt rules and directives for AI crawlers

These rules are examples. Before you deploy them, check the current tokens in each company documentation.

Blocking every bot instead of training only
Before
User-agent: *
Disallow: /
After
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: *
Allow: /

A general group with Disallow: / also shuts out OAI-SearchBot, Claude-SearchBot and Googlebot. Named groups block only the training tokens.

A nosnippet left over from staging
Before
<meta name="robots" content="noindex, nosnippet">
After
<meta name="robots" content="max-snippet:-1, max-image-preview:large">

With max-snippet:-1 Google chooses the snippet length itself. A page with no directives can be indexed and quoted by default.

Common mistakes

Common AI crawler access mistakes

  • One rule for every AI bot

    A block in the User-agent: * group shuts the page off from training and also from AI and classic search engines. Block training bots in named groups.

  • A noindex on a page blocked in robots.txt

    Google says a page with an indexing rule must not be blocked in robots.txt, because the crawler never fetches it and never sees the rule. Removal from results needs access plus noindex.

  • Content added only by JavaScript

    A single-page app without server rendering sends an empty HTML shell. A crawler that skips scripts gets only that shell.

  • Expecting Google-Extended to remove a page from AI Overviews

    Google-Extended does not affect Google Search. You limit content in AI Overviews with nosnippet, max-snippet, data-nosnippet or noindex.

  • Relying on robots.txt for user-triggered bots

    OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them. These bots fetch a page at the request of a specific person.

Numbers and sources

AI crawlers and directives according to the documentation

ValueWhat it meansSource
Independent settingsOpenAI describes GPTBot (training) and OAI-SearchBot (ChatGPT search) as independent, so you can block one and allow the other.OpenAI, crawlers
About 24 hoursHow long OpenAI says it takes for ChatGPT search to reflect a robots.txt change.OpenAI, crawlers
3 botsHow many robots Anthropic describes: ClaudeBot for training, Claude-SearchBot for search and Claude-User for user requests.Anthropic, crawlers
No effectGoogle-Extended does not affect a site inclusion in Google Search and is not a ranking signal.Google, crawlers
nosnippetIn Google it turns off the snippet in all results, including AI Overviews and AI Mode, and stops the content being used as their direct input.Google, robots directives
December 2024When Vercel published an analysis of traffic on its network: none of the major AI crawlers, including GPTBot, ClaudeBot and PerplexityBot, rendered JavaScript.Vercel, The rise of the AI crawler

Sources checked:

Glossary

Terms in the result

robots.txtRobots Exclusion Protocol (RFC 9309)
A file with rules on which pages each crawler may fetch.
User-agentCrawler name
The token a crawler looks for to find its rules in robots.txt.
SSRServer-side rendering
The server sends finished HTML with the content, without waiting for JavaScript.
nosnippetRobots directive
Turns off the snippet in results, and in Google also use in AI Overviews.
JSON-LDJSON for Linked Data
schema.org structured data written in a script tag.
SEO module in Insight

This test checks one thing on one page

One page is a sample. The SEO module in Insight checks AI crawler access in robots.txt and the other signals across the whole site, then repeats the check after a release.