## **🧭 Overview**

When you run a site audit in Search Atlas, our crawler needs to access your website's pages to collect SEO data. If your **robots.txt** file is misconfigured — especially after a WordPress update — it can block our crawler entirely, resulting in incomplete or missing audit results. This article explains why this happens and how to fix it.

## **⚠️ Why Is the Search Atlas Crawler Being Blocked?**

There are several common reasons our crawler cannot access your site:

- **WordPress is set to discourage search engines:** WordPress has a built-in setting that adds a `Disallow: /` directive to your robots.txt, blocking all crawlers.
- **A plugin overwrote your robots.txt:** SEO plugins (e.g., Yoast, Rank Math) or security plugins may have regenerated or modified your robots.txt after an update.
- **A custom robots.txt rule blocks all bots:** A wildcard rule like `User-agent: *` followed by `Disallow: /` blocks every crawler, including ours.
- **Your hosting provider or firewall is blocking crawlers:** Some security configurations or WAF (Web Application Firewall) rules may block unknown user agents.

## **🛠️ How to Fix Robots.txt on WordPress**

1. **Check the WordPress search engine visibility setting.** Log in to your WordPress dashboard. Go to **Settings → Reading**. Make sure the checkbox labelled **"Discourage search engines from indexing this site"** is unchecked. Save your changes. WordPress will update your robots.txt automatically.
2. **Review your robots.txt file directly.** Open a browser and visit **yourdomain.com/robots.txt**. Look for any line that says `Disallow: /` under `User-agent: *`. If you see this, it means all crawlers — including Search Atlas — are blocked.
3. **Edit robots.txt via your SEO plugin.** If you use an SEO plugin such as **Yoast SEO** or **Rank Math**, check your plugin's settings for a robots.txt editing option. Consult your plugin's official documentation for the exact location of this feature, as menu paths may vary by version. Make sure the file contains at minimum: `User-agent: *` followed by `Allow: /`.
4. **Edit robots.txt via FTP or cPanel (advanced).** If you do not use an SEO plugin, connect to your server via FTP or cPanel File Manager. Locate **robots.txt** in your root directory. Remove any `Disallow: /` lines that apply to all user agents. Save and re-upload the file.
5. **Allow the Search Atlas crawler specifically.** If you want to keep other crawlers restricted but allow Search Atlas, you can add a dedicated rule to your robots.txt for the Search Atlas crawler user agent. Place this rule above any wildcard rules to ensure it takes priority. Contact our support team to confirm the exact user-agent string to use.

If you need further assistance, open the chat widget in the bottom-right corner of the platform and type **human teammate** to be connected with a member of our team.