⚠️ Robots.txt "Disallow: /" Warning in the Search Atlas WordPress Plugin

The Search Atlas WordPress plugin flags a Robots.txt "Disallow: /" warning when your robots.txt file contains a rule that blocks all search engine crawlers from your entire website — a single misconfigured line that can silently remove every page from Google search results. Follow the steps below to locate and remove the rule in under two minutes.

📌 Version requirement: This warning requires Search Atlas WordPress plugin version 2.4 or later. A detection bug in earlier versions (fixed May 2026, WP-322) prevented the warning from firing even when Disallow: / was present. Update the plugin before relying on this check.

⚠️ What the "Disallow: /" Warning Means

Your robots.txt file tells search engines which pages they are allowed to crawl. The line Disallow: / under a User-agent directive instructs every crawler — including Googlebot — to stay off your entire website.

The Search Atlas WordPress plugin flags this as a critical warning because one misconfigured line can de-index your whole site without any visible error in WordPress itself.

Common causes include:

• A site left in development or staging mode — WordPress automatically writes Disallow: / when the "Discourage search engines" checkbox is enabled under Settings → Reading.
• A manually edited robots.txt file containing a leftover or accidental rule.
• A third-party plugin or theme that wrote an overly broad rule during initial setup.

Removing the rule restores full crawl access immediately. Previously de-indexed pages may take several days to weeks to reappear in Google search results as Googlebot re-crawls the site.

🔍 How to Find the Source of Your Robots.txt in WordPress

Your robots.txt file is publicly accessible at https://yourdomain.com/robots.txt. Replace yourdomain.com with your actual domain. For WordPress Multisite or subdomain installs, check robots.txt at the root domain level, not the subsite URL. In WordPress, one of three sources controls its content:

• WordPress core setting — Settings → Reading → "Discourage search engines from indexing this site" checkbox.
• SEO plugin — Tools such as Yoast SEO, Rank Math, or All in One SEO generate and manage robots.txt from their own settings panels.
• Physical file — A static robots.txt file in your site's root directory, editable via FTP, cPanel File Manager, or SSH.

Identify which source applies to your site, then follow the matching fix below.

🛠️ How to Fix the "Disallow: /" Warning in WordPress

Choose the method that matches how your robots.txt is managed.

Fix via WordPress Settings (most common)

1. In your WordPress admin dashboard, go to Settings → Reading.
2. Find the checkbox labeled "Discourage search engines from indexing this site".
3. Uncheck the box.
4. Click Save Changes.

WordPress removes the Disallow: / line automatically. Visit https://yourdomain.com/robots.txt in your browser to confirm the line no longer appears.

Fix via an SEO Plugin

1. Open your SEO plugin's robots.txt editor — for example, Yoast SEO → Tools → File Editor, or Rank Math → General Settings → Edit robots.txt.
2. Find any line that reads Disallow: / under a User-agent: * block.
3. Delete the line entirely, or replace it with Disallow: (an empty value) to permit full crawl access.
4. Save the file.

The updated rule takes effect immediately for all new crawl requests.

Fix via a Physical File (FTP or File Manager)

1. Connect to your server using FTP, SSH, or your hosting control panel's File Manager.
2. Open the file robots.txt located in your site's root directory.
3. Delete the line Disallow: /, or change it to Disallow: to allow all crawling.
4. Save and upload the file.

Confirm the change at https://yourdomain.com/robots.txt before re-running the Search Atlas audit.

✅ How to Verify the Fix in the Search Atlas WordPress Plugin

After editing your robots.txt, confirm the warning is cleared directly in the Search Atlas WordPress plugin:

1. In your WordPress admin dashboard, go to OTTO SEO → Site Audit → Overview (Website Overview).
2. Click the Recrawl Site button to start a fresh scan.
3. Open the Robots.txt section in the audit results — the "Disallow: /" warning should no longer appear.

A clean result in the Robots.txt section confirms search engines can crawl your site. If the warning persists, check whether a second source — a plugin or a physical file — is overwriting your robots.txt with the same rule.

💡 Robots.txt Best Practices to Prevent Future Warnings

• Reserve Disallow: / for local or staging environments only — it must never appear in a live production site's robots.txt.
• Use targeted disallow rules to block specific folders (e.g., Disallow: /wp-admin/) rather than blanket restrictions.
• Audit after every site migration. WordPress often retains the "Discourage search engines" checkbox state when a staging site is cloned to production.
• Re-run the Search Atlas Site Audit after major plugin updates, theme switches, or server migrations to catch robots.txt regressions early.

Following these habits protects your site from accidental crawl blocks before they affect your rankings.

🎯 You now know what the Search Atlas "Disallow: /" warning signals, where the rule originates in WordPress, and how to remove it. Re-run the Site Audit to confirm your robots.txt is clean — then explore the Search Atlas Site Audit overview to identify and resolve any other critical SEO issues on your site.