CometWeb-Insight Bot: scans only on request.

CometWeb-Insight is a web audit scanner operated under the CometWeb brand (Maciej Zmitrukiewicz, Poland, no registered company yet). It runs single, contextual audits across five dimensions: performance (Core Web Vitals, synthetic, field), SEO (indexation, sitemap, render), accessibility (WCAG 2.2), security (CSP, headers, TLS), and carbon footprint (SWDM v4).

We only scan addresses someone asked for. Most scans are ordered by signed-in CometWeb Insight customers for sites in their panel: their own, their clients' or competitors they compare against. On top of that come one-off checks of a single address from the free tools on cometweb.io and the free audit. A regular scan does not verify domain ownership, which is why we always honor robots.txt and the scan opt-out form. Ownership proof (a DNS TXT record) is required only for the modes described under “Identification” below.

Standard audits and crawls send the User-Agent below and follow the robots.txt rules for the token below. The exceptions are listed underneath.

Mozilla/5.0 (compatible; CometWeb-Insight/2.1; +https://cometweb.io/bot)

CometWeb-Insight

• Limits are per host (domain), not a global cap across the internet. Other domains may be scanned in parallel for other customer jobs.
• Crawl-delay respected. If your robots.txt sets Crawl-delay: N, we wait N seconds between requests.
• No recurring scans without explicit customer scheduling. We don't scan without a reason.
• Adaptive backoff. 429 / 503 → immediate backoff and a 60-minute scan pause.

• Start: from URLs the customer adds in Insight (not an open-web crawl).
• Internal links: full crawls follow links up to the plan limit (Free 20 · Growth 200 · Studio 500 · Scale 1,000 pages).
• robots.txt and crawl-delay are honored; Disallow paths are not fetched. Only a customer who proved ownership of the domain with a DNS TXT record can switch this off, and only for that domain.
• No auth bypass: we do not log in past auth forms or paywalls (except the customer’s own browser session in Lens).
• Schedule: scans are on-demand; recurring only if the customer enables Monitoring / schedules in the app.

Add the snippet below to your robots.txt. We honor it immediately (cache TTL ≤ 1h). Once blocked, the customer sees "Site blocked CometWeb-Insight in robots.txt" in their panel instead of an analysis.

User-agent: CometWeb-Insight
Disallow: /

You can also block specific paths only:

User-agent: CometWeb-Insight
Disallow: /admin
Disallow: /api/internal

No access to robots.txt, or want scans to stop right now? Use the scan opt-out form: 7 days immediately, 180 days once you confirm through a link sent to an address at your domain.

If your WAF blocks unknown bots, you can whitelist us by User-Agent and, for extra certainty, by the IP addresses we share on request:

• User-Agent contains: CometWeb-Insight/2.1
• IP addresses: on request at hello@cometweb.io (the reverse DNS of our server does not point to cometweb.io)

# Expression
(http.user_agent contains "CometWeb-Insight")

# Action
Allow

Current scanner IP ranges are available on request: email hello@cometweb.io. We host on Hostinger in Frankfurt (Germany, EU).

• robots.txt: we honor Allow / Disallow, Crawl-delay and Sitemap; the one exception is an owner with a verified domain who switched this off for their own crawl.
• GDPR: we scan public site content; no personal data is harvested from page bodies. Full RoPA + DPIA: cometweb.io/legal/gdpr.
• EU AI Act: Limited Risk classification (Art. 50 transparency). AI recommendations are clearly labeled "AI generated" in the customer panel.
• Full compliance docs: cometweb.io/legal/gdpr.

If the crawler overloads your server, sends bad requests, or you have another complaint, email hello@cometweb.io. We usually reply within 1–2 business days. Urgent (5xx / overload): subject URGENT crawl + domain.