AI visibility checker
A free check of one page for AI crawlers, robots.txt for 14 crawlers, text without JavaScript, directives and schema
- Free
- No account required
This AI visibility checker reads one public page the way an AI crawler gets it: the server HTML, robots.txt and the indexing directives, skipping JavaScript. It checks whether AI search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot may fetch the page and how much text is left without JavaScript. It also checks whether noindex or nosnippet limits quoting, and which structured data and llms.txt the site has. The result shows readiness to be read. This tool does not collect AI-system answers or measure citations. Mentions and links in a selected sample of answers need a separate observation, with the question, system and date recorded.
What to fix first
Start with robots.txt. A single Disallow: / under the wrong user agent hides the page from an AI search engine. Allow the search crawlers you care about, and decide about training crawlers separately. You can test each rule in our robots.txt tester.
Then check the text without JavaScript. Open the page source (Ctrl+U) and search for a sentence from your main content. If it is missing there, a crawler that skips scripts misses it too. Server-side rendering or prerendering fixes that for the whole site.
Directives and schema come next. A noindex or nosnippet left over from a staging setup is a common reason a page is missing from results. Structured data and an llms.txt file help machines read the page, with no promise of citations.
This is one page at one moment. To check every page and repeat the check after each release, see AI search readiness in CometWeb Insight.
Check several kinds of page
robots.txt rules and rendering often differ between templates. Check the home page, one article or product and a category page. If only one template has little text in the server HTML, the fix belongs to that template, not to the whole site.
Google Search Central, AI features and your websiteHow to check if AI crawlers can read your page
You check a page for AI crawlers in three places: robots.txt, the HTML the server sends and the robots directives. Five steps show the order to work in.
-
Run the check on one page
Enter the URL of a page you care about, such as a product page. The tool fetches it, robots.txt and /llms.txt, and the result shows the status for each crawler separately.
-
Read the robots.txt rules for each token
A crawler looks for a group with its own name, ignoring case, and otherwise falls back to
User-agent: *. OneDisallow: /in the general group therefore covers every bot without its own group. -
Check the text in the server HTML
Open the page source (Ctrl+U) and search for a sentence from the main content. If it is missing there and appears only after JavaScript runs, a crawler that skips scripts sees an empty page.
-
Check for noindex and nosnippet
Look in
<meta name="robots">and in theX-Robots-Tagheader. In Google,nosnippetalso stops the content being used as direct input for AI Overviews and AI Mode. -
Decide on search and training separately
OpenAI, Anthropic and Google keep search crawlers apart from the ones that collect training data. Write down a decision for each group and set your robots.txt rules to match it.
Example robots.txt rules and directives for AI crawlers
These rules are examples. Before you deploy them, check the current tokens in each company documentation.
User-agent: *
Disallow: /User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: *
Allow: /A general group with Disallow: / also shuts out OAI-SearchBot, Claude-SearchBot and Googlebot. Named groups block only the training tokens.
<meta name="robots" content="noindex, nosnippet"><meta name="robots" content="max-snippet:-1, max-image-preview:large">With max-snippet:-1 Google chooses the snippet length itself. A page with no directives can be indexed and quoted by default.
Common AI crawler access mistakes
-
One rule for every AI bot
A block in the
User-agent: *group shuts the page off from training and also from AI and classic search engines. Block training bots in named groups. -
A noindex on a page blocked in robots.txt
Google says a page with an indexing rule must not be blocked in robots.txt, because the crawler never fetches it and never sees the rule. Removal from results needs access plus noindex.
-
Content added only by JavaScript
A single-page app without server rendering sends an empty HTML shell. A crawler that skips scripts gets only that shell.
-
Expecting Google-Extended to remove a page from AI Overviews
Google-Extended does not affect Google Search. You limit content in AI Overviews with
nosnippet,max-snippet,data-nosnippetornoindex. -
Relying on robots.txt for user-triggered bots
OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them. These bots fetch a page at the request of a specific person.
AI crawlers and directives according to the documentation
| Value | What it means | Source |
|---|---|---|
| Independent settings | OpenAI describes GPTBot (training) and OAI-SearchBot (ChatGPT search) as independent, so you can block one and allow the other. | OpenAI, crawlers |
| About 24 hours | How long OpenAI says it takes for ChatGPT search to reflect a robots.txt change. | OpenAI, crawlers |
| 3 bots | How many robots Anthropic describes: ClaudeBot for training, Claude-SearchBot for search and Claude-User for user requests. | Anthropic, crawlers |
| No effect | Google-Extended does not affect a site inclusion in Google Search and is not a ranking signal. | Google, crawlers |
| nosnippet | In Google it turns off the snippet in all results, including AI Overviews and AI Mode, and stops the content being used as their direct input. | Google, robots directives |
| December 2024 | When Vercel published an analysis of traffic on its network: none of the major AI crawlers, including GPTBot, ClaudeBot and PerplexityBot, rendered JavaScript. | Vercel, The rise of the AI crawler |
Sources checked:
Terms in the result
- robots.txtRobots Exclusion Protocol (RFC 9309)
- A file with rules on which pages each crawler may fetch.
- User-agentCrawler name
- The token a crawler looks for to find its rules in robots.txt.
- SSRServer-side rendering
- The server sends finished HTML with the content, without waiting for JavaScript.
- nosnippetRobots directive
- Turns off the snippet in results, and in Google also use in AI Overviews.
- JSON-LDJSON for Linked Data
- schema.org structured data written in a script tag.
Enter a URL and choose Check page. Our server fetches the page, its robots.txt and /llms.txt, and the result shows which AI crawlers may fetch it, how much text they get without JavaScript and whether a directive limits quoting. To see whether a specific AI system cites your site, ask it the questions your customers ask and look at the sources of the answer.
For AI search and answers: OAI-SearchBot and ChatGPT-User (OpenAI), Claude-SearchBot and Claude-User (Anthropic), PerplexityBot and Perplexity-User (Perplexity), plus Googlebot and Bingbot for classic results. Training crawlers, that is GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot and meta-externalagent, are a separate decision about licensing your content.
OpenAI documents GPTBot (training) and OAI-SearchBot (ChatGPT search) as independent settings, so you can block GPTBot and still allow OAI-SearchBot. Anthropic (ClaudeBot and Claude-SearchBot), Google (Google-Extended does not affect Google Search) and Apple split them the same way.
Most AI crawlers read the HTML the server sends and skip JavaScript. If content appears only after scripts run, as in many single-page apps, such a crawler gets an empty shell. Server-side rendering or prerendering puts the text in the first response.
It is optional. llms.txt is a community proposal, and Google says its AI features need no special AI text files or markup. Adding one takes a few minutes, so we report it as information. Our llms.txt generator builds one from your sitemap.
Yes. Google states that nosnippet and max-snippet:0 also prevent the content being used as direct input for AI Overviews and AI Mode. A positive max-snippet limits the length of a quote, and data-nosnippet excludes only the marked parts.
Structured data helps machines understand what a page is about, but citations and rankings depend on much more than schema. Google says its AI features need no special schema.org markup. Use the types that match the page and keep the JSON valid.
The tool checks readiness: access, readable text and directives. It does not collect ChatGPT answers. Citations and links in answers to selected questions can be checked separately, with the system and date recorded. That sample does not expose the internal index or the complete history of all answers. Site-wide checks are described on the AI search readiness page of CometWeb Insight.
ChatGPT-User and Perplexity-User do not always follow robots.txt, because they fetch a page at a user request. OpenAI says the rules may not apply, and Perplexity says Perplexity-User generally ignores them. For the search crawlers, OAI-SearchBot and PerplexityBot, both companies point to robots.txt as the access control.
Google-Extended does not block AI Overviews, because Google says it does not affect Search. The token covers training and grounding in other Google systems, such as Gemini. You limit content in AI Overviews and AI Mode with Search directives such as nosnippet or max-snippet.
A crawler without its own group in robots.txt follows the User-agent: * group, as RFC 9309 requires. It matches its name regardless of case. If there is no general group either, the crawler may fetch the whole site.
What else might come in handy
This test checks one thing on one page
One page is a sample. The SEO module in Insight checks AI crawler access in robots.txt and the other signals across the whole site, then repeats the check after a release.