π Why Your Site Audit May Be Stalled
If your project is not showing updated analyze stats while other projects update normally, the most common cause is that a firewall or security service on your website is blocking the Search Atlas crawler. When the crawler cannot access your site, the audit cannot complete and your stats stay out of date.
Other common causes include:
- Firewall or CDN blocking: Services like Cloudflare, Sucuri, or a host-level firewall may treat our crawler as a bot and block it.
- robots.txt restrictions: Rules that disallow our user agent can prevent crawling.
- Server issues: Downtime, slow response times, or rate limiting on your server can interrupt the crawl.
- Large site size: Very large sites can take longer to crawl, which may look like a stall.
π§Ύ How to Identify the Search Atlas Crawler
To let the crawler through your firewall, allow the user agent the project actually crawls with. By default that is SearchAtlas Bot (https://www.searchatlas.com). You can change it in Crawl Settings, so do not assume every crawl uses that agent, and do not rely on a fixed Search Atlas IP range β the crawler does not publish one and may use proxies.
Open the project's Crawl Settings and check the User agent dropdown. The SearchAtlas option is noted as SearchAtlas Bot (https://searchatlas.com). Other choices are labeled by name only, and no IP ranges are shown. Copy the user agent you selected; there is no in-product IP list to copy.
π§± Whitelisting the Crawler in Popular Firewalls
Add the Search Atlas crawler to your allow list using the steps for your setup:
- Cloudflare: Go to Security, then WAF, then Create rule. Set the rule to allow traffic where the user agent matches the Search Atlas crawler, then deploy the rule.
- Sucuri or similar WAF: Add the crawler user agent or IP range to your whitelist in the firewall settings panel.
- WordPress security plugins: Look for a bot whitelist or firewall exceptions setting and add the Search Atlas user agent there.
- Host-level firewall: Ask your hosting provider to allow the Search Atlas crawler IP range for your domain.
After updating your firewall, also check that your robots.txt file does not disallow the Search Atlas user agent, as this can block the audit even when the firewall allows access.
π How to Trigger a Fresh Crawl
Once the crawler can access your site, start a new audit:
- Open your project in Search Atlas.
- Go to the Site Audit tool.
- On the audit, click Recrawl Site to run a fresh crawl. If the site has never been crawled, the button says Start first crawl. Recrawl Page only recrawls that page. Use Crawl Settings only to change crawl options; changes apply on the next crawl.
- Wait for the crawl to finish. Larger sites take longer, so allow time before checking results.
- When the crawl finishes, open the project's Site Audit overview and review the updated results, including Site Health and the last crawled time. The figures update only after the crawl completes and can look unchanged if the site itself did not change.
If the crawl still does not complete after whitelisting, run it once more to confirm the firewall change has taken effect, since some firewall rules can take several minutes to propagate.
β οΈ When to Contact Support
If the crawl continues to stall after you have whitelisted the crawler and triggered a new audit, our team can investigate the crawl logs for your project. If you need further assistance, open the chat widget in the bottom-right corner of the platform and send your message. A member of our support team will reply in the chat.
When contacting support, include your project name, the date of the stalled crawl, and a list of the firewall changes you have already made. This helps us resolve the issue faster.