CometWeb-Insight Bot — scans only on request of the site owner.
A web audit scanner operated by CometWeb (sole proprietorship Maciej Zmitrukiewicz, Poland). We scan across five dimensions: performance, SEO, accessibility, security, and carbon footprint — but only sites our customers have explicitly added to their panel.
Who we are
CometWeb-Insight is a web audit scanner operated by CometWeb (sole proprietorship Maciej Zmitrukiewicz, Poland). It runs single, contextual audits across five dimensions: performance (Core Web Vitals, synthetic, field), SEO (indexation, sitemap, render), accessibility (WCAG 2.2), security (CSP, headers, TLS), and carbon footprint (SWDM v4).
Who triggers a scan
We scan only at the request of the site owner. Every scan is an authenticated request from a registered CometWeb-Insight customer who accepted our terms and confirmed ownership of (or authorization for) the target site.
We don't run automated market crawls. We don't index sites for search. We don't sell data harvested from scans. We don't scan sites our customers haven't explicitly added to their panel.
How to identify us
All of our requests carry the User-Agent string and robots.txt token below.
Mozilla/5.0 (compatible; CometWeb-Insight/1.0; +https://cometweb.io/bot)
CometWeb-InsightA different User-Agent (e.g. Googlebot) claiming to be us is not us. Verification: a reverse DNS lookup on our IPs resolves to a CometWeb-controlled domain (*.cometweb.io).
Frequency and intensity
- Crawl-delay respected. If your
robots.txtsetsCrawl-delay: N, we wait N seconds between requests. - No recurring scans without explicit customer scheduling — we don't scan without a reason.
- Adaptive backoff. 429 / 503 → immediate backoff and a 60-minute scan pause.
How to block us
Add the snippet below to your robots.txt. We honor it immediately (cache TTL ≤ 1h). Once blocked, the customer sees "Site blocked CometWeb-Insight in robots.txt" in their panel instead of an analysis.
User-agent: CometWeb-Insight Disallow: /
You can also block specific paths only:
User-agent: CometWeb-Insight Disallow: /admin Disallow: /api/internal
How to whitelist us (Cloudflare / WAF)
If your WAF blocks unknown bots, you can whitelist us by User-Agent and (optionally) reverse DNS:
- User-Agent contains:
CometWeb-Insight/1.0 - Reverse DNS:
*.cometweb.io(after IP lookup)
# Expression (http.user_agent contains "CometWeb-Insight") # Action Allow
Current scanner IP ranges are available on request — email hello@cometweb.io. We host on Hostinger KM8 (Vilnius, Lithuania).
Compliance
- robots.txt — fully respected (Allow / Disallow + Crawl-delay + Sitemap).
- GDPR — we scan public site content; no personal data is harvested from page bodies. Full RoPA + DPIA: cometweb.io/compliance.
- EU AI Act — Limited Risk classification (Art. 50 transparency). AI recommendations are clearly labeled "AI generated" in the customer panel.
- Full compliance docs: cometweb.io/compliance.
Contact
If something's wrong (over-aggressive crawl, malformed request, any complaint) — let us know. We respond within 24 business hours.
You can also request immediate cessation of all scans against your domain — we'll do it per request, without ownership verification. Better over-block than under-block.
"We're the only web audit tool whose /bot page says 'we only scan with permission'. Honest Operator isn't marketing — it's how we built this product."