## **🔍 Overview**

If your OTTO site audit shows a **"Crawl failed. We'll try again soon."** message, or if a crawl appears stuck and never completes, this article explains what causes these failures, what OTTO does automatically, and what steps you can take to resolve the issue.

## **⚠️ What Causes a Crawl Failure?**

OTTO crawl failures can be caused by issues on either the platform side or your website. Common causes include:

- **Temporary server unavailability** — your site was unreachable at the time of the crawl
- **Crawl budget or rate-limiting** — your server blocked or throttled the crawl bot
- **robots.txt restrictions** — crawling is blocked for the user agent OTTO uses
- **Post-processing interruptions** — the crawl completed but data failed to process correctly afterward
- **Platform-side processing errors** — an internal issue prevented the crawl from finishing or handing off results

In most cases, the failure is not caused by something you did wrong. OTTO is designed to detect these failures and retry automatically.

## **🔄 What OTTO Does Automatically**

When a crawl fails, OTTO will attempt to retry the crawl on your behalf. You do not need to take any action immediately. However, if the retry does not resolve the issue within a reasonable time, or if the audit remains stuck, you should follow the manual steps below.

## **🛠️ How to Manually Trigger a Recrawl**

1. Go to **Left sidebar → OTTO SEO → All Sites**.
2. Locate the site showing the crawl failure or stuck audit.
3. Click the **Scan** button associated with that site to trigger a fresh crawl.
4. Monitor the crawl progress under **Left sidebar → OTTO SEO → Crawl Monitoring**.
5. Once the crawl completes, check **Left sidebar → OTTO SEO → All Audits** to confirm audit results have been generated.

## **✅ How to Rule Out Issues on Your End**

Before escalating, check the following on your website:

- **robots.txt** — confirm that crawlers are not blocked. Visit **yourdomain.com/robots.txt** and look for any `Disallow` rules that may apply to all bots or the OTTO crawl agent.
- **Server availability** — confirm your site was online and responding at the time of the crawl failure.
- **Firewall or security plugins** — tools like Cloudflare, Wordfence, or similar services may block automated crawlers. Temporarily whitelist the crawl or add an exception if needed.
- **Site speed and stability** — if your server is slow or overloaded, the crawl may time out. Check your hosting performance during the crawl window.

If none of the above apply and your site appears healthy, the failure is likely on OTTO's side and our team can investigate further.

## **📊 Check Crawl Status in Real Time**

Use **Left sidebar → OTTO SEO → Crawl Monitoring** to view the live status of your crawl. This page shows whether the crawl is in progress, completed, or failed. If the status appears stuck for more than 30 minutes without updating, proceed with a manual recrawl using the steps above.

## **💡 Tips to Prevent Future Crawl Failures**

- Ensure your site is accessible to crawl bots and not behind authentication walls during scheduled crawls.
- Avoid running large server maintenance tasks or deployments during active crawls.
- Keep your **robots.txt** updated and review it after any site changes.
- Check your hosting plan to ensure it can handle crawl traffic without rate-limiting or throttling.

## **🆘 Still Experiencing Issues?**

If you have followed the steps above and your crawl is still failing or stuck, our team can investigate whether the issue is on the platform side and manually trigger reprocessing if needed. If you need further assistance, open the chat widget in the bottom-right corner of the platform and type **human teammate** to be connected with a member of our team.