🔍 What recrawl frequency means
A recrawl frequency is the preferred interval for checking a URL again, such as every 7, 14, or 30 days. It is a scheduling preference, not a guarantee that every URL will be crawled exactly on that date.
🗓️ Why a URL may be overdue
Several factors can delay a recrawl, even when a URL has a frequent schedule:
- Crawl capacity: Available crawl resources are shared across projects and may affect timing.
- Project eligibility: The project or URL must be active and eligible for recrawling.
- Previous crawl results: Errors, blocked access, timeouts, or unavailable pages can prevent a successful recrawl.
- Scheduling delays: Recrawls may run later than the selected interval when queued work or system processing affects execution.
- Configuration changes: A URL may have been added, removed, or updated after the current schedule was created.
⚙️ How schedules are applied
The selected frequency tells Search Atlas how often to prioritize a URL for crawling. It does not reset the URL's crawl date immediately or force a crawl at the exact interval. For example, a 7-day setting means the URL becomes eligible for recurring processing around that interval, subject to availability and access.
Changing a frequency also does not necessarily trigger an immediate crawl. The next run depends on the URL's current crawl state and the platform's scheduling process.
📊 What to check when crawls are delayed
- Open the relevant project and confirm that the project is active.
- Check that the URL is still included in the project and has the expected recrawl frequency.
- Review the URL's latest crawl date and any crawl status or error details.
- Confirm that the page is publicly accessible and does not block Search Atlas or search-engine crawlers.
- Allow additional time if the URL is queued or the latest crawl attempt was unsuccessful.
🚀 Best practices for recurring crawls
- Use shorter intervals for pages that change often or are business-critical.
- Use longer intervals for stable pages to focus crawl capacity where it matters most.
- Keep project settings and URL lists up to date.
- Fix access, redirect, server, and robots.txt issues that prevent successful crawling.
- Review crawl history regularly instead of relying only on the selected frequency.
💬 Need help investigating a URL?
If a URL remains uncrawled well beyond its expected interval, review its project status, crawl history, and access settings first. If you need further assistance, open the chat widget in the bottom-right corner of the platform and type human teammate to be connected with a member of our team.