🔍 What a failed crawl means
OTTO must crawl your site before it can complete a site audit. A repeated failure can come from your website, access restrictions, slow JavaScript rendering, or an issue during OTTO processing. The message “Crawl failed. We’ll try again soon.” does not by itself identify the exact cause.
A failed crawl does not automatically mean the problem is on OTTO’s side. Use the checks below to rule out common website and configuration issues first.
🧭 Check the crawl status
- Open OTTO SEO from the left sidebar.
- Open Crawl Monitoring to review the latest crawl status and timing.
- Open All Audits to confirm whether the audit is still processing, failed, or completed.
- Check whether multiple attempts failed for the same site and whether the failures occurred at similar stages.
Long processing times do not always indicate a failure. Sites that rely heavily on JavaScript may take substantially longer to crawl. Wait for the status to update before starting repeated manual scans.
🌐 Check your website access
- Open the site in a private browser window and confirm it loads without a login.
- Check that the domain and preferred protocol, such as HTTPS, are correct in the project settings.
- Review firewall, CDN, bot protection, and security tools for blocked or challenged crawler requests.
- Confirm that robots.txt, noindex rules, or server permissions are not preventing access to important pages.
- Check server logs for timeouts, repeated 403 or 429 responses, or 5xx errors during the crawl window.
- Make sure the site is not undergoing maintenance or experiencing intermittent downtime.
⚙️ Review OTTO settings
- Go to OTTO SEO → All Sites.
- Select the affected site and confirm its URL and configuration.
- Use Scan to start a new crawl only after checking access and server health.
- Review OTTO SEO → Installation Guide if the site connection or installation has recently changed.
Do not repeatedly start scans while an earlier attempt is still processing. Multiple overlapping attempts can make it harder to identify the original failure and may increase processing time.
📊 How to tell where the issue is
The issue is more likely related to your site when the site is unavailable, protected by a challenge page, returning server errors, blocking crawlers, or taking a long time to respond. It is more likely related to platform processing when the site is publicly reachable, server logs show successful access, several attempts fail in the same way, and other site checks work normally.
Customers may also see a failure when the crawl completes but the handoff to audit processing does not finish. In that case, the crawl itself may have succeeded even though the audit displays a failure. Share the exact status and timestamp when reporting the issue.
📝 What to collect before contacting support
- Site name and domain.
- The exact error message.
- Approximate start and failure times, including your time zone.
- Screenshots from Crawl Monitoring and All Audits.
- Whether the site uses a CDN, firewall, bot protection, or JavaScript-heavy pages.
- Relevant server log results, including status codes, timeouts, or blocked requests.
- Whether a normal browser visit and other crawling tools can access the site.
This information helps the team determine whether the failure is caused by site access, crawl duration, or an OTTO processing stage.
💬 Get help with a confirmed failure
If the site is accessible, access controls are not blocking crawlers, and multiple attempts continue to fail, the issue may require platform investigation. Include the details above so the team can check the crawl and post-processing stages.
If you need further assistance, open the chat widget in the bottom-right corner of the platform and type human teammate to be connected with a member of our team.