🔍 Overview
SiteGround's built-in security firewall can intercept Search Atlas crawler requests and return a challenge page instead of your actual page content. This means your site may appear to crawl successfully inside the Search Atlas dashboard while the crawler is actually receiving a blocked or challenged response — not real content. This article explains why this happens, how to confirm it, and what steps to take to resolve it.
⚠️ Why the Dashboard May Show a Successful Crawl
Search Atlas records a crawl as successful when the server returns a successful HTTP status code. In some cases, SiteGround's firewall may return a response code that the platform interprets as a success — even when the response body is a CAPTCHA or challenge page rather than your actual content. This is a known limitation that our engineering team is aware of and is reviewing.
🛠️ How to Confirm SiteGround Is Blocking the Crawler
- Log in to your SiteGround Site Tools dashboard.
- Review your site's security and traffic logs to look for repeated requests from Search Atlas crawler IP addresses that are being blocked or challenged by the firewall.
- Check your server access logs to see the exact responses being served to the Search Atlas crawler.
- If you are unsure where to find these logs in your SiteGround account, consult SiteGround's own support documentation or contact their support team for guidance on locating blocked traffic records.
⚙️ About the Search Atlas Crawler User-Agent and IP Addresses
Customers have asked us to clarify two specific technical points:
- User-agent value: The Search Atlas crawler identifies itself with a user-agent string that includes SearchAtlas. To confirm the precise value being sent for your site, contact our support team and provide your project name so we can look up the exact crawler configuration for your account.
- IP addresses: Search Atlas uses crawler IP addresses that may need to be whitelisted in SiteGround's firewall. We do not publish the full IP range publicly, as it is subject to change. Our support team can provide the current IP list relevant to your account so you can whitelist them in SiteGround.
✅ How to Whitelist the Search Atlas Crawler in SiteGround
- Contact our support team (see below) and provide your project name and the domain being crawled. We will supply you with the current crawler IP addresses to whitelist.
- Log in to SiteGround Site Tools and navigate to the security or IP management section of your account. If you are unsure of the exact location, SiteGround's support documentation can guide you to the correct panel for your plan.
- Add each Search Atlas crawler IP address provided by our support team to your whitelist and save your changes.
- Trigger a new crawl in Search Atlas and verify that real page content is now being returned.
📋 What to Have Ready When You Contact Support
Because resolving a SiteGround firewall block requires backend investigation, please have the following information ready when you reach out:
- Your project name in Search Atlas
- The domain that is being crawled
- Any error messages or status codes you have observed in your SiteGround logs or in the Search Atlas dashboard
- The date and approximate time when the crawl issue was first noticed
If you need further assistance, open the chat widget in the bottom-right corner of the platform and type human teammate to be connected with a member of our team.