🔍 What a Failed Crawl Means
An OTTO site audit crawl can fail because the website blocks the crawler, access details are incomplete, the site is temporarily unavailable, or a platform processing step encounters an error. A failed crawl does not automatically mean the issue is on your side or on OTTO’s side.
Some crawls also take longer when JavaScript rendering is enabled or when a site is large. During this time, the crawl may appear stuck even though processing is still in progress.
📊 Check the Crawl Status
- In the left sidebar, open OTTO SEO.
- Select Crawl Monitoring to review recent crawl activity and status messages.
- Open All Audits to check whether the audit completed, failed, or is still processing.
- Review the affected site under All Sites and note the latest crawl time and error message.
Messages such as “Crawl failed. We’ll try again soon.” indicate that the platform could not complete the crawl. The message alone may not identify whether the cause is website access, site performance, or an internal processing issue.
⚙️ Check Your Website Setup
- Confirm the website is online and loads normally in a browser.
- Check that robots.txt or security tools are not blocking Search Atlas or OTTO crawlers.
- Review firewall, CDN, bot protection, and rate-limit settings for blocked requests or challenge pages.
- Confirm that required OTTO installation or access settings are complete. Open OTTO SEO → Installation Guide to review the setup.
- Check whether recent hosting, DNS, redirect, authentication, or security changes could affect crawling.
- For JavaScript-heavy sites, allow additional processing time before treating a long-running crawl as failed.
🔄 Retry the Crawl Safely
- Resolve any access or website availability issues first.
- Open OTTO SEO → All Sites.
- Select the affected site and choose Scan.
- Monitor the new attempt in Crawl Monitoring.
Avoid starting many scans in quick succession. Repeated retries can make it harder to identify the original cause and may increase processing time.
🧭 Tell Platform Issues From Site Issues
The issue is more likely related to the website when the site is unavailable, crawler requests are blocked, security challenges appear, or the failure begins after a site configuration change.
The issue may be platform-related when the website is reachable, crawler access is allowed, the same site repeatedly fails without configuration changes, or multiple sites show similar failures at the same time. Internal handoff and post-processing failures can also cause a crawl to fail after the crawling stage has completed.
💬 When to Contact Support
Before contacting support, record the site name, approximate failure time, exact status message, recent website changes, and whether a retry produced the same result. This helps the team determine whether the failure occurred during crawling or later processing.
If you need further assistance, open the chat widget in the bottom-right corner of the platform and type human teammate to be connected with a member of our team.