Search Atlas Help Center

🌐 Fix Your Website Not Globally Crawlable Flag

Open in
View as Markdown Open in ChatGPT Open in Claude
Camilo Aponte

Camilo Aponte

Last updated on Sep 30, 2026

🔍 What Does 'Not Globally Crawlable' Mean?

When Google flags your website as not globally crawlable, it means Googlebot is being blocked from accessing your site from certain regions or entirely. This prevents Google from fully indexing your pages, which can seriously hurt your search visibility and rankings.

This flag typically appears in Google Search Console and signals that something in your site's configuration is restricting crawl access at a global level.

⚠️ Common Causes of This Issue

  • IP-based geo-blocking: Your server or CDN is blocking requests from Google's IP ranges, which are routed through US-based data centers but crawl on behalf of all regions.
  • Robots.txt misconfiguration: A disallow rule is unintentionally blocking Googlebot from crawling key pages or your entire site.
  • Firewall or security rules: Web application firewalls (WAFs) or DDoS protection tools (e.g., Cloudflare) may be flagging Googlebot as suspicious traffic and blocking it.
  • CDN or hosting restrictions: Your content delivery network may have regional access rules that prevent crawlers from non-local IP ranges.
  • HTTP authentication: Password-protected staging environments or pages requiring login block all crawlers by default.
  • VPN or proxy detection: Some security plugins block requests that appear to come from proxy servers or data centers, which includes Googlebot.

🛠️ How to Diagnose the Problem

  1. Check Google Search Console: Navigate to Settings → Crawling in Google Search Console. If the "Not globally crawlable" warning is active, note any additional details provided.
  2. Test your robots.txt: Go to https://yourdomain.com/robots.txt and review the rules. Use Google Search Console's robots.txt Tester to check if Googlebot is being blocked.
  3. Use the URL Inspection Tool: In Google Search Console, use the URL Inspection Tool on your homepage and key pages. Select Test Live URL to see if Googlebot can currently access those pages.
  4. Review your firewall and CDN settings: Log in to your hosting panel, CDN dashboard (e.g., Cloudflare), or WAF settings and check for any rules that block bot traffic or restrict access by IP range or geography.
  5. Check for HTTP authentication: Confirm that no password protection is active on your live site, especially if you recently migrated from a staging environment.

✅ How to Fix the Issue

  • Update robots.txt: Remove any Disallow: / rules that block all crawlers. Make sure the rule targeting Googlebot does not restrict access to important pages. A correctly configured robots.txt should include User-agent: * followed only by specific disallow rules for pages you intentionally want excluded.
  • Whitelist Googlebot in your firewall or CDN: Add Google's verified crawler IP ranges to your allowlist. You can find the current list at developers.google.com/search/apis/ipranges/googlebot.json. In Cloudflare, create a firewall rule that allows requests where the verified bot equals Googlebot.
  • Disable geo-blocking for crawlers: If you use server-side geo-restrictions, configure an exception for verified search engine bots. Work with your hosting provider or developer to implement this safely.
  • Remove HTTP authentication from live pages: Ensure your production environment does not have basic auth enabled. This is a common oversight after launching from a staging setup.
  • Disable aggressive bot-blocking plugins: If you use a WordPress security plugin (e.g., Wordfence, iThemes Security), check that it is not blocking requests from data center IPs, as Googlebot crawls from data centers.

📊 Verify the Fix in Search Atlas

After making changes, use Search Atlas to monitor your site's crawlability and indexing health. Navigate to Left sidebar → Site Metrics (Site Explorer) to review your site's overall performance and spot any remaining technical issues affecting your search visibility.

You should also return to Google Search Console and re-run the URL Inspection Test on your homepage to confirm Googlebot can now access your site. It may take a few days for the flag to clear after Google re-crawls your site.

💡 Prevention Tips

  • Always review your robots.txt before and after a site migration or relaunch.
  • Test crawl access in a staging environment before going live.
  • Set up Google Search Console alerts so you are notified immediately if crawl issues are detected.
  • Regularly audit your firewall and CDN rules to ensure legitimate bots are not accidentally blocked.

🙋 Still Need Help?

If you need further assistance, open the chat widget in the bottom-right corner of the platform and type human teammate to be connected with a member of our team.