πŸ› οΈ Fix a Stalled Crawl and Firewall Whitelist

Camilo Aponte

Camilo Aponte

Last updated on Nov 8, 2026

πŸ” Why analyze stats stay unchanged

Analyze stats update only after Search Atlas finishes a fresh crawl of that project. If this is the only project that looks frozen, the crawl is usually stalled because the site, a firewall, a CDN, or a security plugin is blocking the crawler. Other projects keep updating because their hosts allow the bot through.

A blocked crawl is not a data delay. The auditor never reaches the pages, so scores, issue counts, and page stats stay on the last successful run.

⚠️ Signs the crawl is blocked

  • Analyze stats do not change after you start a new audit.
  • The crawl stays queued, crawling, or failed for a long time.
  • Only one domain is affected while other projects update normally.
  • Your host, Cloudflare, a WAF, or a security plugin shows blocked bot requests.
  • The homepage loads in a browser but returns 401, 403, 429, or a challenge page to bots.

πŸ›‘οΈ Whitelist the Search Atlas crawler

Ask whoever manages hosting or security to allow the Search Atlas crawler before you run the audit again. Use the user agent and IP addresses shown in your project crawl or site audit settings. Do not guess addresses from old tickets.

  1. Open the affected project and note the crawler user agent and IP list in the crawl or audit settings.
  2. In your firewall, WAF, CDN, or security plugin, add those IP addresses to the allow list.
  3. Allow the Search Atlas user agent. Do not require a CAPTCHA, JavaScript challenge, or login for that bot.
  4. If you use Cloudflare or a similar CDN, set a skip or allow rule for those IPs and the user agent on Bot Fight Mode, rate limits, and country or hotlink blocks.
  5. Allow both HTTP and HTTPS, and do not block common crawl paths such as robots.txt, sitemaps, or public pages.
  6. Save the rule, then purge any security cache so the new allow list applies immediately.

Whitelist only the addresses provided in the platform. If the list is not visible in the project, use live chat and ask a teammate for the current crawler details before changing production firewall rules.

πŸš€ Trigger a fresh analyze run

  1. Confirm the homepage and a few inner pages return a normal 200 status to the crawler, not a block or challenge page.
  2. Open the stalled project and start a new site audit or crawl from the analyze area.
  3. Leave the crawl running until it completes. Starting it repeatedly can keep it queued.
  4. When the run finishes, refresh the project. Updated issue counts, page totals, and scores confirm the crawl succeeded.
  5. If stats still match the old run, check host logs for the same block, adjust the allow rule, and run one more audit.

πŸ“ Where to Find It

UI path: Left sidebar β†’ OTTO SEO β†’ Site Audit β†’ Crawl Monitoring

πŸ’‘ Checks that prevent another stall

  • Keep the allow rule active after plugin, CDN, or host updates. Those updates often reset security settings.
  • Avoid rate limits that treat a full-site crawl as an attack.
  • Make sure robots.txt is not disallowing the whole site.
  • Pause uptime or maintenance mode before the audit.
  • Compare this project with a working project. The difference is often one firewall rule on that domain only.

**πŸ’¬ Still not updatingIf you need further assistance, open the chat widget in the bottom-right corner of the platform and send your message. A member of our support team will reply in the chat.

Was this article helpful?