🔍 Fix a Stuck or Slow Site Audit Crawl

🧭 Overview

A Site Audit crawl typically completes within a few minutes to a couple of hours, depending on your site's size and crawl settings. In some cases, crawls can take longer or appear stuck entirely. This article explains the most common causes and how to fix them.

⏱️ How Long Should a Crawl Take?

Crawl duration varies based on several factors:

• Small sites (under 500 pages): Usually 5–30 minutes
• Medium sites (500–5,000 pages): Typically 1–4 hours
• Large sites (5,000+ pages): Can take several hours

If your crawl has been running for more than 24 hours without completing, something is likely preventing it from finishing normally.

⚠️ Common Causes of a Stuck or Slow Crawl

• JavaScript rendering is enabled: Crawling JavaScript-heavy pages is significantly more resource-intensive. When JS rendering is on and the crawl budget is set to a very low number (1–2), crawls can take 17–31 hours or more.
• Low crawl budget setting: A crawl budget of 1–2 pages per second drastically slows progress on larger sites.
• Site errors or redirect loops: Pages that loop, return errors, or are unreachable can cause the crawler to stall on a single URL.
• Server-side blocking: Some servers detect and block automated crawlers, causing requests to hang indefinitely.
• Very large or complex sitemaps: Sitemaps with thousands of URLs or nested structures can slow the initial crawl setup.

🛠️ Step-by-Step Troubleshooting

1. Check your JavaScript rendering setting. If your site does not rely heavily on client-side rendering, disable JS rendering in your audit settings. This is the single most common cause of extremely long crawls.
2. Increase your crawl budget. Navigate to your Site Audit settings and raise the crawl budget above 1–2. A setting of 5–10 is a good starting point for most sites. Higher budgets complete crawls faster.
3. Cancel and restart the crawl. If the crawl has been running for more than 24 hours, cancel it and launch a fresh crawl with the adjusted settings above. Stalled crawls do not self-recover.
4. Verify your site is reachable. Open your website in a browser and confirm it loads without errors, redirects, or login walls. The crawler must be able to access your pages freely.
5. Check for crawl errors in the audit report. After restarting, monitor the audit dashboard for error messages. A message indicating a crawl is failing or looping on a single URL points to a specific page or redirect issue—review that URL directly.
6. Review your robots.txt file. Make sure your robots.txt is not inadvertently blocking the Search Atlas crawler from accessing key sections of your site.
7. Reduce scope for large sites. If your site has tens of thousands of pages, consider limiting the crawl to a specific subdirectory or URL prefix to isolate and test performance before running a full crawl.

✅ Best Practice Settings for Faster Crawls

• Disable JavaScript rendering unless your site requires it for content to load.
• Set your crawl budget to 5 or higher for most sites.
• Keep your sitemap clean and free of redirect chains or broken URLs before initiating a crawl.
• Schedule crawls during off-peak hours to reduce the chance of server-side throttling.

💬 Still Need Help?

If you have tried the steps above and your crawl is still stuck or failing, our team can investigate directly. If you need further assistance, open the chat widget in the bottom-right corner of the platform and type human teammate to be connected with a member of our team.