If you want to run a site audit and monitor your site health using Search Atlas, sometimes our site crawlers get blocked by CDNs.
Content Delivery Networks (CDNs) are a group of servers that help deliver website content faster. If your website relies on a CDN to increase the performance of your website, then it's possible the CDN's firewall will block our crawler from accessing your site.
Resolving this issue requires allowlisting the Search Atlas crawler's user agent so it can reach your site, since our crawler doesn't use static IP addresses. This post will explain how to do so on Cloudflare, as it is the most popular CDN provider for webmasters.
If you are using another CDN, the process is similar, and can still be done by following the steps outlined below.
Search Atlas crawler user agent: Search Atlas Bot (https://www.searchatlas.com/) — use this exact string when creating allowlist rules in Cloudflare, WordFence, or any other firewall or security plugin.
💡 Using OTTO? You can connect your Cloudflare account directly via API token in OTTO Settings → Integrations. This is the preferred method for handling Cloudflare-related access and may resolve crawler access issues without requiring manual firewall changes.
⚙️ Step-by-Step Instructions
🧱 How to Allow Search Atlas's User Agent in Cloudflare
The recommended approach is to create a WAF Custom Rule (or Firewall Bypass Rule) that specifically allows the Search Atlas user agent, rather than disabling bot protection site-wide.
Follow these steps:
Step 1: Log in to your Cloudflare account and select your domain.
Step 2: Go to Security → WAF → Custom rules.
Step 3: Click Create rule and give it a name (e.g., "Allow Search Atlas").
Step 4: Set the field to User Agent, operator to contains, and value to Search Atlas Bot.
Step 5: Set the action to Skip and select the security features to bypass (such as Bot Fight Mode, Super Bot Fight Mode, and Managed Rules).
Step 6: Click Deploy to save the rule.
After performing the above steps, Search Atlas will have access to your website for monitoring while your bot protection remains active for all other traffic.
⚠️ If you must disable Bot Fight Mode entirely (because a Custom Rule isn't available on your plan), be aware this removes bot protection from your entire site. We recommend re-enabling it as soon as the Search Atlas crawl completes.
👉 For more details, see Cloudflare's WAF Custom Rules documentation.
🔐 How to Allow Search Atlas's User Agent in WordFence (WordPress)
Rather than disabling the entire firewall, allowlist the Search Atlas crawler so other protections stay in place.
Step 1: On the WordPress menu, go to WordFence → Firewall → Allowlisted URLs (or Allowlisted Services).
Step 2: Add a new allowlist entry for the user agent Search Atlas Bot (https://www.searchatlas.com/).
Step 3: Click Save Changes.
🧩 How to Allow Search Atlas's User Agent in Stop Bad Bots (WordPress)
Step 1: On the WordPress menu, go to Block Bad Bots → Settings.
Step 2: Go to the Settings tab.
Step 3: Disable the Block All Bots option, or add our user agent Search Atlas Bot (https://www.searchatlas.com/) to the allowed list.
🔐 How to Allow Search Atlas's User Agent in Blackhole for Bad Bots (WordPress)
Step 1: On the WordPress menu, go to Blackhole → Settings.
Step 2: In the Whitelist User Agents section, add the user agent Search Atlas Bot (https://www.searchatlas.com/).
Step 3: Save changes.
🛡️ How to Allow Search Atlas's User Agent in Shield Security (WordPress)
Step 1: On the WordPress menu, go to Shield Security → IP Rules.
Step 2: Use the gear icon for additional options and select Create New User Agent Rule.
Step 3: Add the Search Atlas Bot (https://www.searchatlas.com) user agent to the bypass/whitelist list.
Step 4: Click Create New Rule.
🛠️ Troubleshooting: Cloudflare Not Detected in OTTO
My Cloudflare token is connected in OTTO but Cloudflare still shows as not installed — what should I do?
If you have connected your Cloudflare API token in OTTO settings and validated it, but OTTO still shows Cloudflare as not detected or inactive, this is a known issue our team has investigated. Please contact Support with your domain and token details so we can review the connection on our end and resolve it for you.
✅ Closing Note
Allowing the Search Atlas User Agent ensures your site audits and monitoring run smoothly without interruption from security plugins or CDNs.