## **🧭 Overview**

Search Atlas does not automatically recrawl your site when you make website changes. To see updated audit results, you must trigger a manual recrawl. Additionally, if your WordPress **robots.txt** file is blocking the Search Atlas crawler, your audit will fail or return incomplete data. This article walks you through both fixes.

## **🔄 How Site Audits Work in Search Atlas**

Search Atlas runs site audits based on snapshots taken at the time of the crawl. Unlike Google, the platform does not passively monitor your site for changes. This means:

- Updates you make to your website are **not reflected automatically** in your audit results.
- You must manually trigger a new crawl after making changes to see updated data.
- Results shown in your dashboard represent the state of your site at the time of the last completed crawl.

Waiting for a Google crawl will not update your Search Atlas audit — these are independent processes.

## **🚀 How to Manually Recrawl Your Site**

1. Log in to your Search Atlas account.
2. Navigate to the site audit section of the platform for the project you want to update.
3. Locate the project for the website you have updated.
4. Open the site audit for that project.
5. Look for the option to run a new crawl or trigger a recrawl for that project and select it.
6. Wait for the crawl to complete — this may take a few minutes depending on site size.
7. Once finished, your audit results will reflect the current state of your website.

**Tip:** Run a manual recrawl any time you publish significant content changes, fix technical SEO issues, or update your site structure to keep your audit data accurate.

## **🛠️ Why the Search Atlas Crawler Gets Blocked**

The most common reason a site audit fails to run is that the Search Atlas crawler is being blocked by your website's **robots.txt** file. This happens frequently on WordPress sites, especially after plugin updates or security setting changes.

Signs that your crawler is blocked include:

- The site audit fails to complete or returns no data.
- You see an error indicating the crawler could not access your site.
- Only a very small number of pages are returned in the audit results.

## **⚙️ How to Fix a WordPress robots.txt Crawler Block**

1. Access your WordPress site and navigate to your **robots.txt** file. You can view it directly by going to **yourdomain.com/robots.txt** in your browser.
2. Look for any **Disallow** rules that may be blocking all crawlers or specifically blocking the Search Atlas user agent. A common culprit looks like this: **Disallow: /** under a **User-agent: \*** rule.
3. To edit your robots.txt file in WordPress, access your WordPress dashboard and use an SEO plugin (such as Yoast SEO or Rank Math) that includes a robots.txt editor, or edit the file directly via your hosting file manager or FTP.
4. Remove or update the **Disallow** rule that is blocking crawlers so that the Search Atlas crawler can access your site. If you only want to block specific bots, replace the broad **User-agent: \*** rule with more targeted entries.
5. Save your changes and verify the updated **robots.txt** by revisiting **yourdomain.com/robots.txt** in your browser.
6. Once the block is removed, return to Search Atlas and trigger a manual recrawl as described above to get updated audit results.

**Note:** If you are unsure which plugin or setting is generating your robots.txt rules, temporarily deactivating recently updated security or SEO plugins can help identify the source of the block.

If you need further assistance, open the chat widget in the bottom-right corner of the platform and type **human teammate** to be connected with a member of our team.