🔍 Overview
When the Search Atlas crawler cannot access a Cloudflare-protected site, the recommended resolution is to have your client whitelist the Search Atlas crawler IPs in their Cloudflare configuration. This guide explains what to request from your client and what information to have ready if you need to escalate.
✅ Step 1: Confirm the Crawler Is Actually Being Blocked
Before changing any Cloudflare settings, verify that Cloudflare is the true source of the block. A misconfigured project or incorrect URL can produce the same symptoms.
- In your Search Atlas project settings, confirm the target domain is entered correctly, including any subdomain.
- Ask your client to check their Cloudflare dashboard for blocked requests around the time the crawl was attempted. If Search Atlas IPs appear as blocked, Cloudflare is actively preventing access.
- If no blocks appear in Cloudflare, the issue may be a robots.txt restriction, a server-side firewall, or a login/paywall rather than Cloudflare. Resolve those first before proceeding.
⚙️ Step 2: Review Cloudflare Protection Settings with Your Client
Many crawl failures stem from Cloudflare protection settings that your client can adjust without a full IP whitelist. Work with your client to check whether any bot protection, firewall, or rate limiting configurations are blocking the crawler, and whether temporarily adjusting those settings during a scheduled crawl window is feasible.
📋 Step 3: Request IP Whitelisting from Your Client
If reviewing and adjusting Cloudflare settings is not sufficient or practical, the resolution is to have your client whitelist the Search Atlas crawler IPs in their Cloudflare configuration. Contact our support team using the method below to obtain the current list of Search Atlas crawler IP addresses to share with your client.
📞 Need Help Escalating?
If you are escalating this issue, please have the following ready: the affected project name, the target domain (including any subdomain), and a description of the crawl behavior observed (e.g., fully blocked, partial data returned, specific error messages).
If you need further assistance, open the chat widget in the bottom-right corner of the platform and type human teammate to be connected with a member of our team.