CometWeb tools
Robots.txt tester
Paste robots.txt, choose a crawler and check whether the rules allow it to fetch your URL.
Paste your robots.txt
Copy the file contents and enter the URL you want to test. Analysis runs locally — nothing is uploaded.
This robots.txt tester checks locally in your browser whether pasted rules block a URL for a given crawler. Path matching follows Google’s specification: the longest matching pattern wins, and a tie resolves to Allow. It does not fetch the file from your server — it analyses what you paste.
Why robots.txt blocks too much
The most common mistake is a prefix where a directory was meant. Disallow: /fish also blocks /fishheads and /fish.html — a directory needs the trailing slash.
The second is believing robots.txt removes a page from the index. It does not. It blocks fetching, so the search engine never sees a noindex tag either — the URL can stay in results without a description.
The third is blocking CSS and JavaScript. Without them the search engine renders the page incompletely, which distorts its read of layout and accessibility.
Analysis runs locally in your browser. This tool does not fetch the file from your server and cannot tell you whether a crawler actually obeyed the rules — it shows what the rules say.
Test URLs that should remain accessible
Alongside the blocked directory, test a product page, a category and resources needed for rendering. Repeat for the relevant User-agent. The result depends on the selected bot's rules and path pattern.
Matching rules
- *Wildcard
- Stands for any run of characters. /*.php matches /index.php and /folder/x.php.file.html.
- $End anchor
- Pins the pattern to the end of the path. /*.php$ matches /a.php but not /a.php?x=1.
- LMLongest match
- The longest matching pattern decides, regardless of order in the file.
- AWAllow wins
- On equal pattern length the least restrictive rule wins, which is Allow.
Paste the file contents, enter the full URL and pick a crawler. The tester finds the User-agent group that applies to that crawler, matches the path patterns, and shows the rule that decides the outcome along with its line number.
Yes. Analysis happens in your browser and the pasted contents are not sent to CometWeb. That is also why it does not fetch the file from your server — it checks exactly what you paste.
No. It blocks fetching, not indexing. A blocked URL can still appear in results without a description if links point to it. Excluding a page needs a meta robots noindex or an X-Robots-Tag header — and the page has to be fetchable for the search engine to see either.
Paths in robots.txt match as prefixes, not as whole segments. To block only the directory, add a trailing slash: Disallow: /fish/.
The one with the longer pattern. On equal length Allow wins as the least restrictive.
What else might come in handy
Indexability, metadata and structure — with priorities and a client report.