📊 What Crawl Distribution Means
Crawl distribution shows how many of your site's URLs were crawled and processed within a given time period. The number you see for a month reflects activity inside that specific date range. Because it is tied to a window of time, the figure naturally rises and falls as crawl activity changes from one month to the next.
A drop, such as going from 7,000 in May to 1,000 in June, does not automatically mean something is broken. In most cases there is a clear, expected reason. This article walks you through the common causes and how to check each one.
🗓️ Reason 1: The Month Is Still In Progress
This is the most common cause. If you compare a completed month against one that is only partway through, the current month will always look smaller. May had a full set of days to accumulate crawls, while June may still be gathering data.
- A figure of 1,000 mid-June compared to 7,000 for all of May is usually on track.
- Wait until the month closes before comparing totals side by side.
🔁 Reason 2: Crawl Frequency and Scheduling
Crawls run on a schedule rather than continuously. The number of crawls in a period depends on how often your project is set to crawl and when the last crawl completed. If a large crawl finished late in May, that volume is counted in May, leaving June lower until the next crawl runs.
- Check when your most recent crawl completed.
- Confirm whether a scheduled crawl is still pending for the current month.
⚙️ Reason 3: Project or Setting Changes
Changes to your project configuration can affect how many URLs are crawled. Review whether any of the following changed recently:
- Crawl limits were lowered, reducing the number of URLs processed.
- URL scope or filters were narrowed, excluding pages that were previously included.
- Sitemaps or seed URLs were updated or removed.
- The project was paused or its schedule was adjusted.
📦 Reason 4: Your Monthly Crawled Pages Budget Was Reached
Your plan includes a Number of Monthly Crawled Pages quota. Once the pages crawled in a month reach this budget, further crawling for that period stops until the quota resets at the start of the next month. If May consumed most of the budget late in the month, June can start lower until the allowance refreshes.
- Look for the Crawl Budget Depletion Warning in the platform, which indicates the monthly page budget has been used up.
- Compare the pages crawled against your plan's monthly crawled pages allowance to see if the quota was the limiting factor.
- Note that the depletion warning appears once the budget is already used up. A proactive warning that flags when your crawl settings would exceed the monthly page budget before a crawl runs is currently in development and not yet available.
🌐 Reason 5: Changes On Your Website
Sometimes the drop reflects real changes on your site rather than the platform. Consider whether any of these apply:
- Pages were removed, redirected, or returned errors, leaving fewer URLs to crawl.
- Your robots.txt file or meta directives now block sections of the site.
- Server slowdowns or downtime prevented pages from being reached during the crawl.
🔍 How to Check Your Crawl Data
Follow these steps to confirm what is happening:
- Open the affected project in your dashboard.
- Set the date range to a full, completed month so you compare equal time periods.
- Review the crawl history to see when the last crawl ran and how many URLs it covered.
- Check your crawl settings for recent changes to limits, scope, or scheduling.
- Check whether your monthly crawled pages budget was reached, including any crawl budget depletion warning.
- Verify your robots.txt and sitemap are accessible and unchanged.
✅ When the Drop Is Expected
A lower number is normal when the current month is incomplete, when a large crawl landed in the prior month, when your monthly crawled pages budget was reached, or when you intentionally reduced crawl scope or limits. In these cases no action is needed, and numbers typically even out once the month completes, the quota resets, and the next crawl runs.