🔍 Why Crawl Speed Varies
Site crawls in Search Atlas do not run at a fixed speed. Several factors affect how quickly your pages are processed, and it is completely normal for a crawl to take hours — or even longer — depending on your site's conditions.
- Site size: Larger sites with hundreds or thousands of pages naturally take more time to crawl than smaller ones.
- Server response time: If your web server responds slowly, the crawler waits for each response before moving to the next page. A slow host directly slows down your crawl.
- Platform-wide queue: Crawls from all Search Atlas users are processed through a shared queue. During peak usage periods, your crawl may progress more slowly as system resources are distributed across active jobs.
- Page complexity: Pages with a large number of internal links, redirects, or heavy JavaScript can take longer to fully process.
- Crawl rate limits: To protect your server from overload, the crawler respects rate limits. This prevents large sites from being flooded with rapid requests.
📊 What Is a Normal Crawl Speed?
There is no single "normal" speed because every site and server environment is different. As a general guide:
- Small sites (under 50 pages) may complete within 15–60 minutes.
- Medium sites (50–500 pages) typically take a few hours.
- Larger sites (500+ pages) can take several hours to over a day.
Seeing only 24 out of 110 pages crawled after several hours is within the range of expected behaviour, particularly if your server has slower response times or the platform queue is busy. The crawl is still running — it has not stalled.
⚙️ How to Check Your Crawl Progress
You can monitor an active crawl directly inside Search Atlas:
- In the left sidebar, click Site Metrics (Site Explorer).
- Select your site from the list at /site-explorer/list.
- Review the crawl status indicator to see how many pages have been processed and whether the crawl is still active.
As long as the crawl status shows as in progress, the job is running normally. Refreshing the page periodically will update the page count.
💡 Tips to Help Your Crawl Complete Faster
While you cannot directly control platform queue times, there are steps you can take to reduce delays caused by your own site:
- Improve server response time: Use caching, a CDN, or speak to your hosting provider about performance optimisation. Faster server responses mean faster crawls.
- Fix redirect chains: Multiple redirects on a single URL slow down the crawler. Resolve any unnecessary redirect chains in advance.
- Avoid triggering a re-crawl unnecessarily: Starting a new crawl before a previous one finishes can reset your progress. Wait for the current crawl to complete first.
- Crawl during off-peak hours: Scheduling crawls overnight or during low-traffic periods on your site can sometimes result in faster completion.
🚨 When Should You Be Concerned?
In most cases, a slow crawl is not a problem — it just requires patience. However, you should reach out for support if:
- The crawl status has shown no progress at all for more than 24 hours.
- The crawl appears to be stuck on the same page count over an extended period without any movement.
- You receive an explicit error message or the crawl shows a failed status.
If you are unsure whether your crawl is progressing normally, it is always fine to check in with our team.
🛠️ Need Further Help?
If you need further assistance, open the chat widget in the bottom-right corner of the platform and type human teammate to be connected with a member of our team.