Home / Crawler info
Crawler information
How our scanner identifies itself, the limits it respects, and how to control it.
Last updated: 18 October 2026
About our crawler
When UrlTestly inspects a website, it identifies itself with a clear user agent so site owners can recognize the traffic:
- User agent: Mozilla/5.0 (compatible; UrlTestlyBot/1.0; +https://urltestly.com/crawler)
- Purpose: On-demand safety and reputation analysis requested by a visitor.
- Behavior: A single, lightweight visit per scan. We do not perform large-scale crawling of your site.
Request limits
Every scan operates within strict limits to be a good citizen of the web:
- A short connection and overall request timeout
- A capped number of redirects followed
- A capped amount of HTML downloaded per page
- robots.txt and sitemap reads are size-limited
- Free scans are rate-limited per visitor
Respecting robots.txt
Because a scan is an explicit, user-initiated safety check of a single page, it behaves like a visitor rather than an indexing crawler. We read robots.txt and sitemap declarations, and we never bulk-crawl a site.
Preventing scans of your site
If you operate a site and prefer not to be scanned, you can block our user agent at your edge or CDN. If you believe our crawler is behaving incorrectly, contact [email protected] and we'll investigate promptly.