## **🔍 Understanding the crawl failure**

\*\* When a Site Audit shows “Crawl has failed,” the audit did not complete successfully. The cause may be temporary or related to website access settings, security tools, server problems, or an internal processing issue.

## **✅ Check the crawl details**

1. Open the affected project in Search Atlas.
2. Go to the OTTO site audit or crawl section.
3. Review the latest crawl status and any available error details.
4. Confirm that the crawl is running for the correct domain and protocol, such as HTTP or HTTPS.

## **🌐 Check website availability**

- Open the website in a browser and confirm it loads normally.
- Check whether the site returns server errors such as 403, 429, or 5xx responses.
- Confirm that the domain’s DNS records resolve correctly.
- Make sure the hosting provider has not suspended the site or limited automated requests.

## **🔒 Review security and access settings**

Firewalls, CDNs, bot protection, and security plugins can block audit crawlers. Review these tools for blocked requests or rate limits. If your site uses an allowlist, add the crawler access required by your Search Atlas configuration.

Also check your robots.txt file and any password, maintenance, or login requirement. The pages that need auditing must be publicly accessible to the crawler.

## **⚙️ Retry the crawl**

1. Fix any availability, security, or access issue you find.
2. Save the website changes and confirm the site loads successfully.
3. Start a new crawl from the OTTO audit area.
4. Allow the crawl time to finish before starting another retry.

If the crawl fails again, compare the failure time with server, firewall, CDN, or hosting logs. Repeated failures at the same time can help identify a request block or server limit.

## **📋 Information to include when requesting help**

If the issue continues, note the project domain, the date and time of each failed crawl, the displayed status, and any error details. Also include recent changes to hosting, DNS, robots.txt, CDN, firewall, or security settings.

If you need further assistance, open the chat widget in the bottom-right corner of the platform and send your message. A member of our support team will reply in the chat.