UnblindBot Crawler Documentation
Official technical documentation for webmasters, infrastructure teams, and Cloudflare verified bot registration.
Bot Identity
User-Agent
UnblindBot/1.0 (+https://www.unblind.ai/bot/)
All requests from our crawler include this user-agent for identification.
Purpose
UnblindBot is a read-only SEO analysis crawler operated by unblind.ai. It fetches web pages exclusively on behalf of authenticated customers who have explicitly submitted URLs for analysis through our platform. The crawler is used to:
- Analyze on-page SEO elements (titles, meta descriptions, headings, content)
- Check indexability signals (robots directives, canonical tags, status codes)
- Assess page performance metrics (load time, response headers)
- Extract structured content for AI-powered optimization recommendations
Network Information
IP Address
Our crawler operates from the following IP address:
12.132.98.205
Note: This IP is managed by our hosting provider (IONOS SE). For up-to-date information or to verify a specific request, please contact us at crawler@unblind.ai.
Reverse DNS (rDNS)
All UnblindBot IPs have valid reverse DNS entries pointing to *.unblind.ai domains for authentication purposes.
To verify a request using reverse DNS:
$ dig -x [IP_ADDRESS]
→ crawler.unblind.ai
Hosting Provider
IONOS SE
Elgendorfer Straße 57
56410 Montabaur, Germany
https://www.ionos.com
Crawling Policy
Respectful Crawling
- robots.txt compliance: We honor all robots.txt directives and will not crawl disallowed paths
- Rate limiting: Maximum 1 request per second per domain to avoid overloading servers
- Single fetch policy: Each URL is fetched only once per analysis session
- Caching: Results are cached to minimize redundant requests
- Conservative timeouts: Requests timeout after 30 seconds to avoid hanging connections
- HTTPS only: All requests use secure HTTPS connections
No Circumvention
UnblindBot never attempts to bypass security measures including:
- CAPTCHAs or bot challenges
- Web Application Firewalls (WAF)
- Authentication or login walls
- CDN bot mitigation (Cloudflare, Akamai, etc.)
If our crawler is blocked by your infrastructure and you wish to allow it, our customers receive instructions to request whitelisting from their IT team.
Privacy & Data Usage
We only crawl publicly accessible content that customers explicitly submit for analysis. Crawled content is:
- Stored securely in our database for analysis purposes
- Only accessible to the customer who requested the analysis
- Never shared with third parties
- Subject to our privacy policy
Blocking UnblindBot
If you wish to prevent UnblindBot from crawling your site, you can block it using standard methods:
Method 1: robots.txt
Add the following to your /robots.txt file:
User-agent: UnblindBot
Disallow: /
Method 2: Block by User-Agent
Configure your web server or WAF to block requests with the UnblindBot user-agent.
Method 3: Block by IP Range
Block the IP ranges listed above in your firewall or CDN configuration.
Cloudflare Verified Bot Registration
UnblindBot is registered for Cloudflare's Verified Bots program. This page serves as our public documentation for verification purposes.
Verification Methods Supported
- Public IP List: See Network Information section above
- Reverse DNS (rDNS): All IPs resolve to *.unblind.ai
- HTTP Message Signatures: Coming soon (2025 priority method)
For Cloudflare administrators: This page provides all required information for verified bot registration as per Cloudflare's verified bots documentation.
Contact & Support
For questions, concerns, or reports of abusive behavior from our crawler:
Email: crawler@unblind.ai
General contact: hello@unblind.ai
Response time: Within 48 hours on business days
We take crawler behavior seriously and will promptly investigate any reports of policy violations.
About unblind.ai
unblind.ai is a SaaS platform for AI-powered SEO analysis and optimization. Our crawler is an essential component of our service, enabling us to provide accurate, actionable insights to e-commerce teams and digital marketers.
Learn more about unblind.ai