πŸ” Fix Search Atlas Crawler Sitemap Access Issues

Camilo Aponte

Camilo Aponte

Last updated on Nov 13, 2026

πŸ“Œ Overview

Sometimes Site Audit reports that a sitemap is inaccessible, even though it worked before. Possible causes include a changed sitemap address, a security rule, a redirect, or a temporary fetch problem. This article explains how to check common causes. (content-assistant/app/utils/domain\_analysis\_utils.py:292–300, 358–365)

❓ What You May See

  • A message that no sitemap was found or the sitemap is not accessible.
  • Pages that were found in earlier crawls no longer appear in recent crawls.
  • The sitemap opens fine in your browser, but the crawler still reports a problem.

🧭 Why This Happens

  • The sitemap address is slightly different. Common names are sitemap.xml, sitemap_index.xml, and wp-sitemap.xml. A name like sitemaps.xml (with an extra s) is easy to mistype and may not exist on your site.
  • The sitemap moved or was renamed. An SEO plugin update, a theme change, or a site migration can change the sitemap address.
  • A security tool is blocking the crawler. Firewalls, CDNs, and bot protection can block automated visitors, even when the page loads normally for you.
  • The sitemap returns an error. The file might return a 404 (not found), 403 (forbidden), or 5xx (server error) message.
  • Too many redirects. A redirect from http to https, or from non-www to www, can cause problems if it loops or sends the crawler to the wrong place.
  • The file is not valid XML. Broken formatting, extra characters, or an error message inside the file can stop it from being read.
  • robots.txt is blocking access. Rules in your robots.txt file may block the sitemap or the crawler.

πŸ› οΈ Step-by-Step Troubleshooting

  1. Confirm the exact sitemap address. Open your site's robots.txt file (for example, yourdomain.com/robots.txt) and look for a line starting with Sitemap. This shows the official address of your sitemap.
  2. Open the sitemap in a private browser window. If it shows a 404 or error page, the address is wrong or the file is missing. Update it to the correct address.
  3. Check that the page loads as XML. You should see a list of URLs or a list of other sitemap files. If you see a normal web page, a login page, or an error message, fix that on your website first.
  4. Check the final address after redirects. Make sure the sitemap loads in one step on your preferred version of the site (https and your chosen www or non-www version).
  5. Review your robots.txt rules. Make sure there is no Disallow rule covering your sitemap file or blocking all crawlers from your site.
  6. Check firewall, CDN, and security plugin settings. Look for bot protection, rate limiting, or country blocking that may stop the crawler. If you find blocked requests, allow the Search Atlas crawler and then run a new crawl.
  7. Check your SEO plugin or CMS settings. Make sure sitemap generation is switched on and that the file is being created. Re-save your settings or regenerate the sitemap if needed.
  8. Check the sitemap address on your website. If your robots.txt file references the sitemap, update that reference to the correct full URL, including https:// where applicable. Then run a new crawl.
  9. Run a new crawl. After making changes, start a fresh crawl and check whether the sitemap is now found.

βœ… Quick Checklist

  • The sitemap address is correct and matches what robots.txt lists.
  • The sitemap opens without a login and shows a successful result.
  • There are no redirect loops.
  • The file is valid XML and not empty.
  • robots.txt does not block the sitemap.
  • Your firewall or CDN is not blocking the crawler.
  • The correct address is saved in your Search Atlas project settings.

πŸ’‘ Tips to Prevent This Issue

  • Use one standard sitemap address and avoid changing it often.
  • List your sitemap in robots.txt so crawlers can find it automatically.
  • Recheck your sitemap after plugin updates, redesigns, or migrations.
  • Review security and firewall settings after any change that affects bot traffic.
  • Keep your sitemap limited to live pages that return a normal status.

πŸ’¬ Still Need Help?

If the sitemap still cannot be accessed after these checks, please have the following ready: your project name, the full sitemap address, and the date of the most recent crawl that failed. This helps our team investigate faster.

If you need further assistance, open the chat widget in the bottom-right corner of the platform and send your message. A member of our support team will reply in the chat.

Was this article helpful?