🧭 Overview
Search Atlas does not automatically recrawl your site when you make website changes. To see updated audit results, you must trigger a manual recrawl. Additionally, if your WordPress robots.txt file is blocking the Search Atlas crawler, your audit will fail or return incomplete data. This article walks you through both fixes.
🔄 How Site Audits Work in Search Atlas
Search Atlas runs site audits based on snapshots taken at the time of the crawl. Unlike Google, the platform does not passively monitor your site for changes. This means:
- Updates you make to your website are not reflected automatically in your audit results.
- You must manually trigger a new crawl after making changes to see updated data.
- Results shown in your dashboard represent the state of your site at the time of the last completed crawl.
Waiting for a Google crawl will not update your Search Atlas audit — these are independent processes.
🚀 How to Manually Recrawl Your Site
- Log in to your Search Atlas account.
- Navigate to the site audit section of the platform for the project you want to update.
- Locate the project for the website you have updated.
- Open the site audit for that project.
- Look for the option to run a new crawl or trigger a recrawl for that project and select it.
- Wait for the crawl to complete — this may take a few minutes depending on site size.
- Once finished, your audit results will reflect the current state of your website.
Tip: Run a manual recrawl any time you publish significant content changes, fix technical SEO issues, or update your site structure to keep your audit data accurate.
🛠️ Why the Search Atlas Crawler Gets Blocked
The most common reason a site audit fails to run is that the Search Atlas crawler is being blocked by your website's robots.txt file. This happens frequently on WordPress sites, especially after plugin updates or security setting changes.
Signs that your crawler is blocked include:
- The site audit fails to complete or returns no data.
- You see an error indicating the crawler could not access your site.
- Only a very small number of pages are returned in the audit results.
⚙️ How to Fix a WordPress robots.txt Crawler Block
- Access your WordPress site and navigate to your robots.txt file. You can view it directly by going to yourdomain.com/robots.txt in your browser.
- Look for any Disallow rules that may be blocking all crawlers or specifically blocking the Search Atlas user agent. A common culprit looks like this: Disallow: / under a User-agent: * rule.
- To edit your robots.txt file in WordPress, access your WordPress dashboard and use an SEO plugin (such as Yoast SEO or Rank Math) that includes a robots.txt editor, or edit the file directly via your hosting file manager or FTP.
- Remove or update the Disallow rule that is blocking crawlers so that the Search Atlas crawler can access your site. If you only want to block specific bots, replace the broad User-agent: * rule with more targeted entries.
- Save your changes and verify the updated robots.txt by revisiting yourdomain.com/robots.txt in your browser.
- Once the block is removed, return to Search Atlas and trigger a manual recrawl as described above to get updated audit results.
Note: If you are unsure which plugin or setting is generating your robots.txt rules, temporarily deactivating recently updated security or SEO plugins can help identify the source of the block.
If you need further assistance, open the chat widget in the bottom-right corner of the platform and type human teammate to be connected with a member of our team.