⏱️ Manual Site Crawl Duration and Speed Settings Explained

🔍 Overview

When you trigger a manual site crawl in Search Atlas, the process can take anywhere from a few minutes to up to 24 hours, depending on your site's size and server conditions. This article explains what to expect, what the crawl speed setting actually does, and why your crawl duration is influenced by factors beyond that setting.

⏳ How Long Does a Manual Crawl Take?

There is no single fixed duration for a manual crawl. The time it takes depends on several variables, but the ranges below give you a reliable reference point:

• Small sites (under 500 pages): Typically completes within 15–60 minutes.
• Medium sites (500–10,000 pages): Usually takes 1–6 hours.
• Large sites (10,000+ pages): Can take anywhere from 6 to 24 hours.

If your crawl has been showing as loading for several hours but falls within the range above for your site size, this is expected behaviour. A crawl is only considered stuck if it has exceeded 24 hours without completing.

⚙️ What Does the Crawl Speed Setting Actually Control?

The crawl speed setting controls the rate at which the crawler sends requests to your server — measured in requests per second. In simple terms, it determines how aggressively the crawler asks your site for pages.

• Lower speed: Fewer requests per second, which is gentler on your server and reduces the risk of triggering rate limits or temporary blocks.
• Higher speed: More requests per second, which can help the crawler move through pages faster — but only when your server can respond quickly enough to keep up.

Think of crawl speed as setting the maximum pace the crawler tries to move at — not the pace it will always achieve.

🚧 Why Doesn't Crawl Speed Guarantee Total Duration?

Even with the highest crawl speed selected, your total crawl time can still be long. This is because crawl speed only controls one part of the process. Several server-side and backend factors determine the actual time to completion:

• Server response times: If your web server is slow to respond to requests — due to hosting limitations, high traffic, or page complexity — the crawler must wait before it can move on. No speed setting overrides this.
• Rate limiting and throttling: Some servers automatically slow down or temporarily block rapid requests. When this happens, the crawler backs off to avoid being blocked, regardless of your speed setting.
• Redirect chains and crawl depth: Sites with deep page hierarchies or long redirect chains require more individual requests to fully map, extending total time.
• Backend processing time: After the crawler collects raw data, Search Atlas processes and analyses that data on the backend. This processing phase runs after the crawl itself and adds to the total time before results appear.
• Total number of unique URLs: The more unique pages discovered during a crawl, the longer the full process takes, independent of speed.

📋 What to Do If Your Crawl Seems Stuck

1. Check how long the crawl has been running. If it is under 24 hours, allow it to continue — this is within the expected window.
2. Verify that the site being crawled is publicly accessible and not returning widespread errors (such as 5xx server errors), which can cause the crawler to stall.
3. If the crawl has exceeded 24 hours with no change in status, try cancelling and restarting the crawl from OTTO SEO → Site Audit → Crawl Monitoring.
4. Consider lowering the crawl speed setting if your server has limited capacity — this can prevent throttling and actually result in a more stable, complete crawl.

💡 Tips for Faster, More Reliable Crawls

• Schedule manual crawls during low-traffic periods on your site so the server can respond more quickly.
• Use a moderate crawl speed if you are on shared hosting — very high speeds can trigger server-side blocking that slows the overall crawl.
• Ensure your robots.txt file is not inadvertently blocking the Search Atlas crawler from key sections of your site.
• If you only need data for a specific section, consider whether a targeted crawl of a subdirectory is more efficient than a full-site crawl.

🙋 Need Further Help?

If you need further assistance, open the chat widget in the bottom-right corner of the platform and type human teammate to be connected with a member of our team.