## **🔍 Overview**

If your live **robots.txt** file contains **Disallow** entries that do not appear in the Search Atlas robots.txt editor, those rules are almost certainly being added by your **hosting provider, WordPress plugin, or another third-party tool** — not by Search Atlas. This article explains why this happens and what you can do about it.

## **⚠️ Why Extra Disallow Entries Appear**

Search Atlas generates and previews the robots.txt rules that it controls. However, your website's final, publicly served robots.txt file is assembled by your hosting environment. Common sources of extra **Disallow** entries include:

- **Hosting provider defaults** — Many managed hosts (e.g. WP Engine, Kinsta, SiteGround) automatically inject their own robots.txt rules, sometimes blocking AI crawlers or staging paths.
- **WordPress plugins** — SEO plugins such as Yoast SEO, Rank Math, or All in One SEO can override or append to the robots.txt file independently of Search Atlas.
- **CMS-level settings** — Some CMS platforms have a built-in robots.txt editor in their admin dashboard that takes priority over other tools.
- **Server-level configuration** — A physical `robots.txt` file stored in your site's root directory will override virtually generated rules.

Because Search Atlas only previews the rules it manages, any entries injected upstream will not appear in the Search Atlas editor — but they will appear when you fetch the live file directly.

## **🛠️ How to Identify the Source**

1. Open your browser and go to **https://yourdomain.com/robots.txt** to see the live file.
2. Compare that output with what is shown in the Search Atlas robots.txt editor.
3. Any **Disallow** lines present in the live file but absent from the Search Atlas preview are coming from an external source.
4. Log in to your hosting control panel and check for a robots.txt management section or firewall/crawl settings.
5. In your WordPress dashboard (if applicable), check **Settings → Reading** and any active SEO plugin for robots.txt controls.
6. Use FTP, SFTP, or your host's file manager to check whether a physical **robots.txt** file exists in your site's root (`/public_html/` or equivalent).

## **✅ How to Remove the Unwanted Entries**

Once you have identified the source, follow the appropriate steps below.

- **Hosting provider rules:** Contact your host's support team and ask them to remove or disable any automatic robots.txt injection. Some hosts provide a toggle in their dashboard under crawl or bot-management settings.
- **Physical robots.txt file:** If a static file exists in your root directory, delete it or edit it to remove the unwanted **Disallow** lines. Search Atlas can then manage the file virtually.
- **WordPress SEO plugin:** Open the plugin's settings, locate its robots.txt editor, and remove any conflicting entries. If you want Search Atlas to be the sole manager, disable the plugin's robots.txt feature entirely.
- **CMS built-in editor:** Navigate to your CMS robots.txt settings and clear any rules that conflict with your Search Atlas configuration.

## **🔄 Verifying the Fix**

1. After making changes, wait a few minutes for your server cache to clear.
2. Reload **https://yourdomain.com/robots.txt** in your browser.
3. Confirm that the unwanted **Disallow** entries are gone.
4. Return to Search Atlas and verify that the preview matches the live file.

## **💡 Best Practices**

- Designate **one tool only** as the robots.txt manager for your site to avoid conflicts.
- After any hosting migration or plugin update, re-check your live robots.txt file to catch newly injected rules early.
- Keep a record of what each **Disallow** entry is for so you can quickly spot anything unexpected in the future.

## **🙋 Need More Help?**

If you have followed the steps above and are still seeing unexpected **Disallow** entries in your live robots.txt file, If you need further assistance, open the chat widget in the bottom-right corner of the platform and type **human teammate** to be connected with a member of our team.