## **🔍 Why your crawl may be blocked**

Site Auditor needs to access your website to collect updated analysis data. A firewall, Web Application Firewall (WAF), bot-protection service, hosting security rule, or server restriction may block the crawler. When this happens, your project may continue showing older statistics or fail to start a new analysis.

## **🛡️ Allow Site Auditor through your firewall**

1. Open your firewall, WAF, CDN, or hosting security settings.
2. Review blocked requests and security events for the time of the Site Auditor crawl.
3. Allow legitimate Site Auditor crawler requests instead of blocking them as bots.
4. Reduce bot protection or challenge requirements for the crawl, if your security provider supports an exception.
5. Make sure your server allows requests from external crawlers and does not require a browser-only challenge, login, CAPTCHA, or JavaScript verification.
6. Confirm that your robots.txt file and server rules do not block the pages you want to analyze.

Use the security provider’s verified crawler or request details when creating an allow rule. Avoid allowing all traffic or disabling your firewall globally.

## **🚀 Trigger a fresh analysis**

1. Sign in to Search Atlas and open the affected project.
2. Open Site Audit for the project. Crawl controls are Recrawl or Recrawl Site, and Crawl Settings. On a project card, Recrawl is also in the card menu.
3. Start a new crawl with Recrawl or Recrawl Site. If a crawl is already running, the button shows Crawling… or Recrawling and will not start another one. Crawl Settings changes the crawl configuration; it does not start the crawl.
4. Wait for the crawl to finish before checking the updated statistics.

If the crawl does not begin, check whether the project has a crawl limit, an incomplete setup, or a website connection issue. Confirm that the project URL is correct and that the site is available publicly.

## **📊 Check whether the crawl completed**

Review the crawl status and completion time in Site Auditor. Updated statistics may not appear until the crawl has fully finished. If the crawl completes but the numbers still look unchanged, compare the crawl date with the date of your latest website changes.

For persistent issues, review your firewall logs while starting another crawl. A new blocked request can identify the rule that needs adjustment.

## **💡 Prevent future crawl interruptions**

- Keep a dedicated firewall exception for approved Site Auditor crawler traffic.
- Monitor WAF rules after security policy changes or website migrations.
- Check robots.txt and access rules whenever your domain, CDN, or hosting provider changes.
- Do not require authentication or interactive verification on pages that Site Auditor must analyze.
- Schedule crawls during a period when your server can handle the additional requests.

If you need further assistance, open the chat widget in the bottom-right corner of the platform and send your message. A member of our support team will reply in the chat.