📅 What the crawl frequency means
A scheduled crawl frequency tells Search Atlas how often a URL should be considered for crawling. For example, a setting of every 7 days means the URL is eligible to be crawled approximately every 7 days. It is not a guaranteed completion time.
⏳ Why a crawl may be delayed
A URL can go beyond its selected frequency when crawl work is delayed or cannot be completed. Common reasons include:
- Other scheduled crawl jobs are being processed first.
- Temporary platform capacity limits or traffic bursts require crawl requests to be spread out.
- The previous crawl was unsuccessful or did not complete.
- The URL is being checked as part of another crawl or processing workflow.
- Results and related recommendations are still waiting for downstream processing.
These delays can cause a URL set to every 7–14 days to show a last crawl date from 30–90 days ago.
⚙️ Frequency settings are not normally removed
If you manually set a recrawl frequency, the setting should remain active. A missed crawl does not mean the frequency was intentionally deactivated. The next eligible crawl may occur after the delayed work is processed.
🔎 How to check a delayed URL
- Open the URL or project area where the crawl schedule is managed.
- Confirm that the URL still has the expected frequency, such as every 7 or 14 days.
- Review the last crawl date and any available crawl status or error details.
- Check whether the URL is included in another active crawl workflow.
- Allow time for queued work to complete, then refresh the crawl status.
✅ What you should expect
Scheduled crawling is designed to prioritize recurring checks while managing crawl capacity safely. The selected frequency is the intended cadence, but actual crawl completion can vary. A delayed crawl does not necessarily indicate a problem with your URL or settings.
🛠️ When to request help
Contact support if a URL remains unprocessed for an extended period, its frequency changes unexpectedly, or the crawl shows a persistent error. Include the project, URL, selected frequency, last crawl date, and any visible error message so the issue can be reviewed efficiently.
If you need further assistance, open the chat widget in the bottom-right corner of the platform and type human teammate to be connected with a member of our team.