🔍 Why This Error Appears in Search Atlas
When Search Atlas crawls your site, it flags a crawl-delay directive in your robots.txt file as a critical error. This directive instructs search engine bots to pause between requests, which can significantly slow down how quickly Google and other search engines index your pages.
If your store runs on Shopify, you have likely already discovered that you cannot simply open and edit your robots.txt file the way you would on a self-hosted site. This is expected behavior — it is a platform-level restriction, not a problem with your account or Search Atlas.
🛍️ Why Shopify Locks Your robots.txt File
Shopify automatically generates a robots.txt file for every store. On most plans, this file is managed by Shopify's infrastructure and is not directly editable through the Shopify admin panel or theme editor by default. The crawl-delay directive you see is typically injected by Shopify itself or by a third-party app installed on your store.
Because of this, standard advice such as "open your robots.txt file and delete the line" does not apply to Shopify stores. You need to use one of the platform-specific methods described below.
✅ Method 1: Check for Apps Injecting the Directive
The most common cause of a crawl-delay directive on Shopify is a third-party SEO or security app that modifies your robots.txt output. Follow these steps to identify and remove the source:
- Log in to your Shopify Admin panel.
- Go to Apps in the left sidebar.
- Review any installed SEO, site speed, or security apps such as SEO Manager, Plug in SEO, or bot-protection tools.
- Open each relevant app's settings and look for a robots.txt or crawl settings section.
- If you find a crawl-delay setting, disable it or set the value to 0 and save.
- Wait 24–48 hours, then re-run a crawl in Search Atlas to confirm the error is resolved.
If you are unsure which app is responsible, temporarily disable SEO-related apps one at a time and check your live robots.txt file at yourdomain.com/robots.txt after each change to identify the source.
✅ Method 2: Use a Custom robots.txt Template (Shopify 2.0 Themes)
If your store uses a Shopify Online Store 2.0 theme, Shopify allows you to override the default robots.txt file using a Liquid template. This method gives you full control over the file's content.
- In your Shopify Admin, go to Online Store → Themes.
- Next to your active theme, click Actions → Edit code.
- In the left file panel, look for a folder called Templates.
- If a file named robots.txt.liquid already exists, click it to open it. If it does not exist, click Add a new template, select robots.txt from the dropdown, and click Create template.
- Review the template content and remove any line that contains crawl-delay.
- Click Save.
- Visit yourdomain.com/robots.txt to confirm the directive is no longer present.
Important: When creating a custom robots.txt.liquid template, Shopify replaces its automatically generated file entirely with your template. Make sure the template retains all necessary Disallow and Sitemap directives from the original. You can copy the current content of your live robots.txt as a starting point before making edits.
✅ Method 3: Contact Shopify Support
If your theme is not a 2.0 theme, the robots.txt.liquid template option is unavailable, and no installed app appears to be the source of the directive, the crawl-delay may be hard-coded by Shopify at the infrastructure level. In this case, direct editing is not possible without Shopify's involvement.
Contact Shopify Support directly and provide the following information:
- Your store URL and the current content of your robots.txt file.
- The specific crawl-delay line you need removed.
- A note that the directive is causing indexation issues flagged by your SEO tooling.
Shopify support can advise whether an upgrade to a 2.0-compatible theme or a platform-level change is required.
📊 Verifying the Fix in Search Atlas
After applying any of the methods above, confirm the error has been resolved in the Search Atlas dashboard:
- In the left sidebar, navigate to OTTO SEO V3.
- Open your active project and locate the site audit or technical SEO section.
- Trigger a new crawl of your site.
- Once the crawl completes, confirm the crawl-delay critical error no longer appears in your issue list.
If the error persists after the crawl, revisit your robots.txt file directly at yourdomain.com/robots.txt to verify the directive has been fully removed before re-crawling.
💬 Need More Help?
If you need further assistance, open the chat widget in the bottom-right corner of the platform and type human teammate to be connected with a member of our team.