🔍 Why Your Site Audit Crawl Gets Stuck
If your Site Audit crawl has been running for more than 24 hours without completing — or shows no progress at all — SiteGround's built-in security tools are almost certainly blocking the Search Atlas crawler. This is one of the most common reasons WordPress sites hosted on SiteGround fail to crawl successfully.
SiteGround uses two layers of protection that can silently block automated crawlers:
- SiteGround Security plugin — flags and bans IP addresses that make rapid, repeated requests (exactly what a crawler does).
- Cloudflare integration — SiteGround routes traffic through Cloudflare by default, which can trigger bot-protection rules that block crawl requests before they even reach your server.
Because these blocks happen silently, your crawl appears to be running but never actually retrieves any pages.
✅ Before You Start: Confirm the Issue
Check these two things first to confirm a block is the cause:
- Your Site Audit project was added recently and the crawl has not completed after 24+ hours.
- Your WordPress site is hosted on SiteGround (check your hosting dashboard or domain registrar if unsure).
If both are true, follow the steps below in order.
🛠️ Step 1 — Whitelist the Crawler in the SiteGround Security Plugin
If you have the SiteGround Security plugin installed on your WordPress site, it may have automatically banned the Search Atlas crawler IP range. Here is how to allow it:
- Log in to your WordPress admin dashboard.
- In the left menu, go to Security → Blocked IPs.
- Look for any recently blocked IP addresses. If you see entries added around the time your crawl started, these are likely the Search Atlas crawler IPs.
- Select those entries and click Unblock.
- Next, go to Security → Activity Log Settings and add the Search Atlas crawler user agent (SearchAtlasBot) to your allowlist so it is not blocked in future crawls.
If you are unsure which IPs belong to Search Atlas, open the chat widget and a teammate can provide the current IP ranges.
☁️ Step 2 — Adjust Cloudflare Bot Protection
SiteGround enables Cloudflare by default for many hosting plans. Cloudflare's bot-fight mode or firewall rules can block the crawler independently of the SiteGround Security plugin.
- Log in to your SiteGround hosting dashboard.
- Go to Speed → Cloudflare and click Manage next to your domain.
- Inside Cloudflare, navigate to Security → Bots.
- If Bot Fight Mode is enabled, temporarily disable it and restart your crawl to test whether this resolves the issue.
- Alternatively, go to Security → WAF → Firewall Rules and create an allow rule for the SearchAtlasBot user agent so Cloudflare does not challenge or block it.
Note: You can re-enable Bot Fight Mode after whitelisting the Search Atlas user agent. Disabling it permanently is not necessary or recommended.
🔄 Step 3 — Restart Your Site Audit Crawl
After completing the steps above, restart the crawl from Search Atlas:
- In the left sidebar, go to OTTO SEO → Site Audit → Overview (Website Overview).
- Find your affected project.
- Click Recrawl Site (or delete and re-add the project if the crawl appears frozen).
- Monitor progress over the next 15–30 minutes. Most sites complete an initial crawl within this window once the block is lifted.
💡 Tips to Prevent This in the Future
- Whitelist SearchAtlasBot before adding new projects — if you host multiple sites on SiteGround, whitelist the crawler user agent proactively in both the SiteGround Security plugin and Cloudflare.
- Check your SiteGround Activity Log regularly — the plugin logs blocked IPs in real time, making it easy to spot false positives.
- Avoid aggressive rate-limiting rules — if you have custom Cloudflare firewall rules that throttle requests per IP, consider excluding known SEO tool bots from those rules.
💬 Still Seeing Issues?
If your crawl is still stuck after following all the steps above, there may be a server-level firewall rule or a custom .htaccess restriction that requires closer inspection. If you need further assistance, open the chat widget in the bottom-right corner of the platform and type human teammate to be connected with a member of our team.