## **🗺️ Overview**

When OTTO runs a site audit, it crawls your website to collect SEO data. Occasionally, this crawl may fail or return incomplete results. This article explains the most common causes of crawl failures, how to tell whether the issue is on OTTO's side or yours, and the steps to get your crawl running again.

## **🔎 How to Check Crawl Status in OTTO**

1. Go to **Left sidebar → OTTO SEO → All Sites**.
2. Locate the site you want to investigate and check its status indicator.
3. For a more detailed view, navigate to **Left sidebar → OTTO SEO → Crawl Monitoring**.
4. Review the crawl log entries for error messages, timestamps, and any noted interruptions.

The **Crawl Monitoring** page is your first stop for diagnosing what went wrong. It shows whether the crawl started, how far it got, and where it stopped.

## **⚠️ Common Reasons a Crawl Fails**

Crawl failures can originate either on OTTO's side or on your website's side. Use the list below to identify which applies to your situation.

- **Site was unreachable during the crawl:** If your server was down, in maintenance mode, or returning 5xx errors at the time OTTO attempted to crawl, the crawl will fail. Check your server logs for the crawl timestamp.
- **Robots.txt is blocking the crawler:** If your `robots.txt` file disallows all bots or specifically blocks OTTO's user agent, pages will not be crawled. Review your robots.txt at **yourdomain.com/robots.txt**.
- **Password protection or authentication walls:** If your site requires a login, HTTP authentication, or is behind a firewall, OTTO cannot access pages without credentials.
- **Crawl budget exceeded or timeout:** Very large sites may time out before the crawl completes. This appears as a partial or failed crawl in Crawl Monitoring.
- **OTTO installation issue:** If the OTTO script is not correctly installed, audit features may not function as expected. Verify your setup via **Left sidebar → OTTO SEO → Installation Guide**.
- **Temporary platform-side issue:** On rare occasions, a crawl may fail due to a transient issue within Search Atlas. These are typically resolved quickly and a recrawl will succeed.

## **✅ How to Rule Out Issues on Your End**

Follow these steps to confirm your website is accessible and correctly configured for crawling:

1. **Check server uptime:** Confirm your site was online at the exact time the crawl was attempted. Cross-reference with your hosting provider's uptime logs.
2. **Review robots.txt:** Visit **yourdomain.com/robots.txt** and confirm no rules are blocking all crawlers or OTTO specifically.
3. **Disable password protection temporarily:** If your site is behind a login or IP restriction, disable it briefly and trigger a recrawl to test access.
4. **Verify OTTO installation:** Go to **Left sidebar → OTTO SEO → Installation Guide** and confirm the OTTO script is correctly placed on your site.
5. **Test with an external crawler:** Use a tool like Google Search Console's URL Inspection to confirm Google can also access your pages. If Google is blocked, OTTO likely is too.

## **🔄 How to Trigger a Recrawl**

Once you have identified and resolved the issue, you can manually start a new crawl:

1. Go to **Left sidebar → OTTO SEO → All Sites**.
2. Select the site you want to recrawl by checking its checkbox.
3. Click the **Scan** button in the top action bar.
4. Navigate to **Left sidebar → OTTO SEO → Crawl Monitoring** to watch the crawl progress in real time.

If the crawl completes successfully, your updated audit data will appear under **Left sidebar → OTTO SEO → All Audits**.

## **📋 What to Do If the Crawl Keeps Failing**

If you have checked all of the items above and the crawl still fails, gather the following information before reaching out to support:

- The domain name of the affected site.
- The date and time of the failed crawl (visible in Crawl Monitoring).
- Any error messages shown in the crawl log.
- Confirmation of whether your robots.txt and server access have been verified.

If you need further assistance, open the chat widget in the bottom-right corner of the platform and type **human teammate** to be connected with a member of our team.