spippbot

Ukurasa huu unapatikana kwa Kiingereza na Kijerumani.

spippbot is the crawler of spipp.com. User agent: spippbot/0.1 (+https://spipp.com/bot).

Rules

spippbot respects robots.txt (RFC 9309) including Crawl-delay, meta robots (noindex, nofollow), rel=nofollow and canonical links. By default it fetches at most one page every 2 seconds per host and backs off further on errors (429, 5xx).

spippbot does not bypass logins, paywalls or CAPTCHAs and does not access private network addresses.

Opting out

Add “User-agent: spippbot” and “Disallow: /” to your robots.txt to exclude your whole site.