⚙️ Whitelist Search Atlas Crawler IPs in Your CDN or Firewall

If your site is protected by a CDN, firewall, or security service, Search Atlas crawlers may be blocked from accessing your website. When this happens, Site Audit and other crawling features may fail to collect data or report incomplete results.

This article explains how to allow Search Atlas crawlers through your security layer.

🤔 Why This Happens

Many Content Delivery Networks (CDNs), Web Application Firewalls (WAFs), and hosting security providers automatically block unknown bots or limit requests from crawler IP addresses.

If Search Atlas crawler IPs are blocked, you may experience:

• ❌ Site Audits that fail or never complete.
• ❌ Missing crawl data.
• ❌ Incomplete site health reports.
• ❌ Pages not being discovered during scheduled crawls.

✅ Solution

Allow all official Search Atlas crawler IP addresses in your CDN, firewall, or hosting provider.

You can always find the most up-to-date IP list here:

👉 https://bots.searchatlas.com/ip_list.txt

⚠️ Important

Do not manually maintain your own IP list. Always use the official Search Atlas list, as crawler IPs may be added or updated over time.

☁️ If You're Using Cloudflare

Cloudflare makes this even easier.

1. Log in to your Cloudflare dashboard.
2. Open your website.
3. Navigate to your Bot Management or Security settings.
4. Select Search Atlas from the Verified Bots list.
5. Save your changes.

📘 If needed, refer to Cloudflare's documentation for additional information about Verified Bots.

🛡️ Other CDNs or Firewalls

If you're using another provider (such as Akamai, Fastly, Imperva, Sucuri, AWS WAF, or another security platform):

1. Open your firewall or bot management settings.
2. Whitelist all IP addresses from the official Search Atlas IP list.
3. Save your configuration.
4. Run your Site Audit again.

🔍 Verify the Fix

After updating your security settings:

1. Open Site Audit in Search Atlas.
2. Start a new crawl.
3. Wait for the crawl to begin.

✅ If the crawl starts successfully, your whitelist has been configured correctly.

🚨 Still Having Issues?

If you've already whitelisted the official Search Atlas IP addresses but crawls are still failing:

• ✔️ Verify that your CDN changes have fully propagated.
• ✔️ Check whether another firewall, hosting provider, or security plugin is blocking requests.
• ✔️ Confirm you're using the latest IP list from the official Search Atlas source.
• ✔️ Contact Search Atlas Support if the issue persists.

📋 Quick Summary