🔍 What Is the Crawl-Delay Issue?
If your Search Atlas dashboard is flagging a crawl-delay directive as a critical error in your robots.txt file, you are not alone. This is a common issue for Shopify store owners. Shopify automatically inserts a crawl-delay: 1 directive into every store's robots.txt file — and it cannot be removed or edited by the store owner.
The directive looks like this inside your robots.txt file:
- User-agent: *
- Crawl-delay: 1
This means Shopify is instructing all bots to wait 1 second between each page request when crawling your site. While this protects server performance, it can slow down how quickly search engines index your content.
⚠️ Why Can't I Delete It?
Shopify does not allow merchants to directly edit the robots.txt file in the same way a WordPress or custom-built site would. The platform generates this file automatically, and the crawl-delay line is hard-coded into Shopify's default template. Even if you use Shopify's robots.txt customisation feature (available on certain plans), fully removing the crawl-delay directive is not straightforward and depends on your Shopify plan and theme setup.
This is a platform-level limitation, not a misconfiguration on your part.
📊 Does Crawl-Delay Actually Hurt My SEO?
The impact depends on your site size and your crawl budget:
- Small stores (under 500 pages): The 1-second crawl-delay is unlikely to cause meaningful indexing delays. Googlebot and other major crawlers often ignore or deprioritise this directive.
- Large stores (500+ pages, frequent updates): A crawl-delay can slow down how quickly new or updated pages are discovered and indexed, which may affect time-sensitive content like product launches or price changes.
- Google specifically: Google has publicly stated that Googlebot does not support the crawl-delay directive. However, other bots (Bingbot, and various third-party crawlers) do respect it.
Search Atlas flags this as a critical error because it represents a technical SEO risk that should be reviewed — even if the real-world impact varies by site.
🛠️ What Can You Do About It?
While you cannot fully remove the directive on standard Shopify plans, there are steps you can take to manage and mitigate the issue:
- Verify the directive is present: Go to yourdomain.com/robots.txt in your browser and confirm the crawl-delay line exists. This helps you understand exactly what search engines are seeing.
- Check your Shopify plan: Shopify Plus merchants have access to a liquid-based robots.txt template that allows more granular customisation. If you are on Shopify Plus, work with a developer to modify the robots.txt.liquid template and remove or override the crawl-delay directive.
- Submit your sitemap manually: Compensate for slower crawling by ensuring your XML sitemap is submitted in Google Search Console and Bing Webmaster Tools. This gives search engines a direct map of your pages regardless of crawl speed.
- Use Google Search Console's URL Inspection tool: For high-priority pages (new products, landing pages), use the Request Indexing feature in Google Search Console to prompt faster indexing without relying on a crawl.
- Monitor crawl coverage regularly: Inside Search Atlas, use the OTTO SEO section (left sidebar → OTTO SEO → All Sites (SEO Automation)) to track crawl errors and indexing status over time. This helps you catch any pages that are falling behind on indexation.
✅ How to Mark This Error as Acknowledged in Search Atlas
If you have reviewed the issue and determined the impact is low for your store size, you can acknowledge the error inside Search Atlas to keep your dashboard clean while you monitor the situation:
- Navigate to the site audit section of your Search Atlas dashboard.
- Locate the crawl-delay critical error in your robots.txt findings.
- Review the recommendation details and confirm you have followed the mitigation steps above.
- Use the available acknowledgement or dismiss option if your workflow allows for it, and add a note referencing this article for your records.
Acknowledging the error does not fix the underlying directive, but it signals that your team is aware and has taken appropriate action given the platform constraints.
💡 Key Takeaways
- The crawl-delay directive in Shopify robots.txt is added automatically by Shopify and cannot be removed on standard plans.
- Google ignores the crawl-delay directive, but other bots respect it.
- The real-world SEO impact is low for small stores but worth monitoring for large or frequently updated catalogues.
- Shopify Plus merchants can customise the robots.txt.liquid file to remove the directive.
- Submitting your sitemap and using URL inspection tools in Google Search Console are the best workarounds for all Shopify plans.
If you need further assistance, open the chat widget in the bottom-right corner of the platform and type human teammate to be connected with a member of our team.
Additional Notes
Method 1: Check for Apps Injecting the Directive — third-party SEO/security apps (e.g. SEO Manager, Plug in SEO) may inject crawl-delay. Disable apps one at a time, check live robots.txt, or find crawl settings within the app and set value to 0. Wait 24–48 hours then re-crawl in Search Atlas.