π What This Issue Looks Like
You start a Site Audit in Search Atlas, but the crawl never starts, stops after a few pages, or finishes with errors or very little data. This often happens when the website sits behind a security service such as Cloudflare. These services are designed to stop automated visitors, and they can mistake our crawler for an unwanted bot.
This is common when you audit a client's website and don't manage its hosting or security settings yourself.
π¦ Why a Crawl Gets Blocked
A site audit works like a search engine visitor. It requests your pages one after another to check them for issues. Security tools may block those requests when they:
- Come from an IP address the tool doesn't recognize
- Arrive quickly, one request after another
- Don't look like a normal web browser
- Trigger a security rule such as Bot Fight Mode, a managed challenge, or Under Attack Mode
When the crawler is blocked, the site returns an error or a verification page instead of the real content, so the audit can't read the pages.
π§ Signs Cloudflare May Be the Cause
- The audit fails right away or shows no pages crawled
- Only one or a few pages are crawled
- You see errors such as 403 (forbidden), 429 (too many requests), 503 (service unavailable), or a challenge page
- The site loads fine in your browser but not in the audit
- Other crawling tools also have trouble with the same site
β Quick Checks You Can Do First
Before changing any security settings, rule out the simple causes:
- Check the URL. Make sure the address matches the live version of the site, including http or https and www or non-www. Open it in your browser to confirm it loads without redirect loops.
- Check that the site is public. Staging sites, password-protected pages, and sites behind a login can't be crawled.
- Check the robots.txt file. Visit yourdomain.com/robots.txt and look for rules that disallow all crawlers, such as a line with Disallow: /.
- Check for other security tools. Hosting firewalls and security plugins, such as those used with WordPress, can block crawlers just like Cloudflare.
- Try again. Temporary server slowdowns can cause a failed crawl. Restart the audit once to confirm the problem is consistent.
π‘οΈ How to Confirm a Cloudflare Block
If you or your client can log in to Cloudflare, check whether it is stopping our requests:
- Log in to the Cloudflare dashboard and select the website.
- Open the Security section and go to Events.
- Look for blocked or challenged requests that match the time you started the audit.
- Open an entry to see which rule acted on the request, such as a WAF rule, a bot setting, or a rate limit.
If you see entries from the time of your crawl, Cloudflare is the likely cause. Menu names may vary slightly depending on your Cloudflare plan and dashboard version.
βοΈ How to Allow the Crawler
The safest fix is to create an exception for the Search Atlas crawler rather than turning off security. Ask the site owner or developer to do one of the following:
- Add a custom rule: In the Security settings, create a rule that allows or skips security checks for requests from the Search Atlas crawler.
- Allow the crawler's IP addresses: Add the crawler's IP addresses to the IP Access Rules or allow list.
- Review bot protection: If Bot Fight Mode or Super Bot Fight Mode is turned on, check whether it is challenging the crawler. An exception may be needed.
- Review rate limiting: If rate limiting rules are active, make sure they aren't blocking the crawler for sending many requests in a short time.
- Turn off Under Attack Mode during the audit: This mode shows a verification page to every visitor, which the crawler can't complete. Turn it on again after the audit if needed.
To get the current crawler details to allow, such as its IP addresses and identifying information, send a message in the chat widget described at the end of this article.
π After You Make Changes
- Save the changes in Cloudflare and wait a few minutes for them to apply.
- Return to Search Atlas and start a new Site Audit for the website.
- Check that the number of crawled pages is close to what you expect.
If the audit still fails, review the Cloudflare Security Events again. A different rule may be blocking the crawler, and each rule needs its own exception.
π‘ Tips for Agencies Working on Client Sites
- Ask your client for access to Cloudflare, or send their developer the steps in this article.
- Request the allow rule before you start a new client project so your first audit runs smoothly.
- Keep a record of which clients have allowed the crawler so you can troubleshoot faster later.
- Don't ask a client to disable all security. A targeted exception is safer and works just as well.
π¬ Still Need Help?
If you need further assistance, open the chat widget in the bottom-right corner of the platform and send your message. A member of our support team will reply in the chat.
To help us resolve your issue faster, include the website address, the time you started the audit, and any error messages or screenshots from Cloudflare.