Search Atlas Help Center

🔍 Fix Crawler Blocking and Understand Crawl Scheduling

Open in
View as Markdown Open in ChatGPT Open in Claude
Camilo Aponte

Camilo Aponte

Last updated on Sep 30, 2026

🗺️ Overview

Search Atlas uses its own crawler to audit your site independently of Google. This means two separate processes are at work: the Search Atlas crawler (which powers your site audit data inside the platform) and Google's crawler (which determines your rankings). Understanding the difference — and making sure Search Atlas can actually access your pages — is the first step to accurate audit results.

⏱️ When Do Search Atlas Results Update After Site Changes?

Search Atlas does not update automatically the moment you publish changes. Here is how the process works:

  1. You trigger or schedule a crawl. Navigate to your site audit dashboard inside the platform and start a fresh crawl manually using the crawl option available there.
  2. The Search Atlas crawler visits your pages. Depending on your site size, this can take a few minutes to several hours.
  3. Your audit data refreshes. Once the crawl completes, all metrics — broken links, missing tags, page speed issues — reflect the current state of your site.

This process is entirely separate from Google. Even after Google re-indexes your updated pages, your Search Atlas audit data will only reflect those changes after a new Search Atlas crawl is run.

🚫 Why Is the Search Atlas Crawler Being Blocked?

If you see an error message indicating the crawler is blocked, it means your site's robots.txt file is preventing Search Atlas from accessing your pages. This is one of the most common reasons audit data appears incomplete or stale. A blocked crawler cannot collect data — regardless of what the platform status shows.

Do not assume your site is configured correctly if you have received a blocking error. The error message is definitive. Follow the steps below to diagnose and fix the issue.

🔎 How to Check Your robots.txt File

Your robots.txt file lives at the root of your domain. To view it, type the following into your browser address bar, replacing the example with your own domain:

https://yourdomain.com/robots.txt

Look for any lines that contain Disallow rules that could block the Search Atlas crawler. A blocking rule would look like one of these:

  • User-agent: * followed by Disallow: / — this blocks all crawlers, including Search Atlas.
  • A user-agent entry for the Search Atlas crawler followed by Disallow: / — this blocks Search Atlas specifically.
  • Disallow: / appearing under a wildcard rule with no corresponding Allow exception for the Search Atlas crawler.

🛠️ Fixing robots.txt Blocking on WordPress

On WordPress, your robots.txt rules are often controlled by an SEO or security plugin rather than a static file. Follow these steps to locate and resolve the blocking rule:

  1. Check your SEO plugin settings. If you use a plugin such as Yoast SEO or Rank Math, navigate to its settings in your WordPress admin dashboard and look for a robots.txt editor. Review any Disallow rules and remove or adjust any rule that blocks all bots or specifically blocks the Search Atlas crawler.
  2. Check your security plugin settings. Security plugins such as Wordfence include bot-blocking features that can prevent crawlers from accessing your site. In the plugin's settings, review any bot or crawler blocking rules and add an exception to allow the Search Atlas crawler through.
  3. Check for maintenance mode. If your site is in maintenance mode (via a plugin or theme setting), all external crawlers will typically be blocked. Disable maintenance mode and re-check your robots.txt file.
  4. Verify the fix. After making changes, revisit https://yourdomain.com/robots.txt in your browser to confirm the blocking rule has been removed. Then return to the Search Atlas platform and trigger a new crawl from your site audit dashboard so the crawler can re-access your pages.
  5. Confirm audit data updates. Once the new crawl completes, verify that your site audit data is now populating correctly. If the crawler-blocked error persists after following these steps, note your domain, the exact error message shown in the platform, and which plugin(s) you checked so our team can assist you efficiently.

If you need further assistance, open the chat widget in the bottom-right corner of the platform and type human teammate to be connected with a member of our team.