If you want to run a site audit and monitor your site health using Search Atlas, sometimes our site crawlers get blocked by CDNs.
Content Delivery Networks (CDNs) are a group of servers that help deliver website content faster. If your website relies on a CDN to increase the performance of your website, then it’s possible the CDN’s firewall will block our crawler from accessing your site.
Resolving this issue requires turning off bot-blocking features, as Search Atlas’s crawler doesn’t use static IP addresses. This post will explain how to do so on Cloudflare, as it is the most popular CDN provider for webmasters.
If you are using another CDN, the process is similar, and can still be done by following the steps outlined below.
⚙️ Step-by-Step Instructions
⚙️ Step-by-Step Instructions
🧱 How to Allow Search Atlas’s User Agent in Cloudflare
Follow these steps:
Step 1: Log in to your Cloudflare account.
Step 2: Click Security and select the Bots submenu.
Step 3: Toggle off either Bot Fight Mode or Super Bot Fight Mode.
After performing the above steps, Search Atlas will have access to your website for monitoring.
👉 For help and a better understanding of Cloudflare’s IP Access Rules, see Cloudflare’s support documentation.
🔐 How to Allow Search Atlas’s User Agent in WordFence (WordPress)
🔐 How to Allow Search Atlas’s User Agent in WordFence (WordPress)
Step 1: On the WordPress menu, go to WordFence → Firewall → Firewall Options.
Step 2: Set the Web Application Firewall Status to Disabled.
Step 3: Click the Save Changes button.
🧩 How to Allow Search Atlas’s User Agent in Stop Bad Bots (WordPress)
🧩 How to Allow Search Atlas’s User Agent in Stop Bad Bots (WordPress)
Step 1: On the WordPress menu, go to Block Bad Bots → Settings.
Step 2: Go to the Settings tab.
Step 3: Disable the Block All Bots option, or add our user agent Search Atlas Bot (https://www.searchatlas.com/) to the allowed list.
🔐 How to Allow Search Atlas’s User Agent in Blackhole for Bad Bots (WordPress)
🔐 How to Allow Search Atlas’s User Agent in Blackhole for Bad Bots (WordPress)
Step 1: On the WordPress menu, go to Blackhole → Settings.
Step 2: In the Whitelist User Agents section, add the user agent Search Atlas Bot (https://www.searchatlas.com/).
Step 3: Save changes.
🛡️ How to Allow Search Atlas’s User Agent in Shield Security (WordPress)
🛡️ How to Allow Search Atlas’s User Agent in Shield Security (WordPress)
Step 1: On the WordPress menu, go to Shield Security → IP Rules.
Step 2: Use the gear icon for additional options and select Create New User Agent Rule.
Step 3: Add the Search Atlas Bot (https://www.searchatlas.com) user agent to the bypass/whitelist list.
Step 4: Click Create New Rule.
✅ Closing Note
Allowing the Search Atlas User Agent ensures your site audits and monitoring run smoothly without interruption from security plugins or CDNs.