CometWeb-Insight Bot — scans only on request of the site owner.

CometWeb-Insight is a web audit scanner operated under the CometWeb brand (Maciej Zmitrukiewicz, Poland — no registered company yet). It runs single, contextual audits across five dimensions: performance (Core Web Vitals, synthetic, field), SEO (indexation, sitemap, render), accessibility (WCAG 2.2), security (CSP, headers, TLS), and carbon footprint (SWDM v4).

We scan only at the request of the site owner. Every scan is an authenticated request from a registered CometWeb-Insight customer who accepted our terms and confirmed ownership of (or authorization for) the target site.

All of our requests carry the User-Agent string and robots.txt token below.

Mozilla/5.0 (compatible; CometWeb-Insight/2.1; +https://cometweb.io/bot)

CometWeb-Insight

• Limits are per host (domain), not a global cap across the internet — other domains may be scanned in parallel for other customer jobs.
• Crawl-delay respected. If your robots.txt sets Crawl-delay: N, we wait N seconds between requests.
• No recurring scans without explicit customer scheduling — we don't scan without a reason.
• Adaptive backoff. 429 / 503 → immediate backoff and a 60-minute scan pause.

• Start: from URLs the customer adds in Insight (not an open-web crawl).
• Internal links: full crawls follow links up to the plan limit (Free 20 · Growth 200 · Studio 500 · Scale 1,000 pages).
• robots.txt and crawl-delay are honored; Disallow paths are not fetched.
• No auth bypass: we do not log in past auth forms or paywalls (except the customer’s own browser session in Lens).
• Schedule: scans are on-demand; recurring only if the customer enables Monitoring / schedules in the app.

Add the snippet below to your robots.txt. We honor it immediately (cache TTL ≤ 1h). Once blocked, the customer sees "Site blocked CometWeb-Insight in robots.txt" in their panel instead of an analysis.

User-agent: CometWeb-Insight
Disallow: /

You can also block specific paths only:

User-agent: CometWeb-Insight
Disallow: /admin
Disallow: /api/internal

If your WAF blocks unknown bots, you can whitelist us by User-Agent and (optionally) reverse DNS:

• User-Agent contains: CometWeb-Insight/2.1
• Reverse DNS: *.cometweb.io (after IP lookup)

# Expression
(http.user_agent contains "CometWeb-Insight")

# Action
Allow

Current scanner IP ranges are available on request — email hello@cometweb.io. We host on Hostinger KM8 (Vilnius, Lithuania).

• robots.txt — fully respected (Allow / Disallow + Crawl-delay + Sitemap).
• GDPR — we scan public site content; no personal data is harvested from page bodies. Full RoPA + DPIA: cometweb.io/legal/gdpr.
• EU AI Act — Limited Risk classification (Art. 50 transparency). AI recommendations are clearly labeled "AI generated" in the customer panel.
• Full compliance docs: cometweb.io/legal/gdpr.

If the crawler overloads your server, sends bad requests, or you have another complaint — email hello@cometweb.io. We reply on business days, 09:00–17:00 CET. Urgent (5xx / overload): subject URGENT crawl + domain.