🧭 Overview
Search Atlas Site Audit crawls your website by following the URLs listed in your XML sitemap. If your sitemap does not include all of your pages, Search Atlas will only crawl the pages it can find there — even if those pages exist on your live site and are accessible to other tools. This is the most common reason customers see a lower crawled page count than expected.
📋 How Search Atlas Discovers Pages
Unlike some crawlers that spider every link on a page, Search Atlas primarily relies on your submitted XML sitemap to discover and audit pages. This means:
- Pages listed in your sitemap will be crawled and audited.
- Pages not listed in your sitemap will be skipped, even if they are live and indexable.
- A third-party tool that crawls by following internal links may report a higher page count simply because it uses a different discovery method.
If you have 500+ pages on your site but only 114 appear in your sitemap, Search Atlas will audit approximately 114 pages — not 500+.
✅ Step 1 — Check How Many URLs Are in Your Sitemap
Before investigating anything else, verify your sitemap is complete:
- Open your sitemap directly in a browser. The most common location is yourdomain.com/sitemap.xml.
- Count the total number of
<url>entries, or use a free sitemap checker tool to get an accurate count. - Compare that number to the total number of pages you expect to be crawled.
- If the sitemap count is significantly lower than your total page count, your sitemap is incomplete — this is the root cause.
🛠️ Step 2 — Fix an Incomplete Sitemap
If your sitemap is missing pages, you need to regenerate or update it so all intended pages are included. How you do this depends on your platform:
- WordPress: Use an SEO plugin such as Yoast SEO, Rank Math, or All in One SEO. Navigate to the plugin's sitemap settings and ensure all post types, pages, and taxonomies you want crawled are enabled. Save and regenerate the sitemap.
- Shopify: Shopify generates a sitemap automatically. If products or pages are missing, confirm they are published and not set to hidden.
- Custom or headless sites: Work with your developer to ensure your sitemap generation script includes all relevant URLs and is not filtering out pages unintentionally.
- Large sites with sitemap indexes: If your site has more than 50,000 URLs, you may need a sitemap index file that references multiple individual sitemaps. Confirm all child sitemaps are linked from the index.
After updating your sitemap, re-submit it in Google Search Console under Sitemaps to keep your data consistent across tools.
🔄 Step 3 — Re-Submit Your Sitemap in Search Atlas
Once your sitemap is updated and includes all the pages you want audited, re-run your Site Audit in Search Atlas:
- Navigate to your project in Search Atlas.
- Open Site Audit and locate the sitemap or crawl settings for your project.
- Confirm the correct sitemap URL is entered. If you have updated the sitemap URL or added a sitemap index, update this field accordingly.
- Start a new crawl and wait for it to complete.
- Verify the crawled page count now matches the number of URLs in your updated sitemap.
⚠️ Other Reasons Pages May Be Missing
If your sitemap is complete but pages are still not appearing in your crawl results, check the following secondary causes:
- Noindex tags: Pages with a
<meta name="robots" content="noindex">tag or an X-Robots-Tag: noindex HTTP header may be excluded from audit results intentionally. - robots.txt blocks: Open yourdomain.com/robots.txt and confirm that the Search Atlas crawler or the Disallow rules are not blocking sections of your site.
- Canonical tags pointing elsewhere: Pages with a canonical tag pointing to a different URL may be treated as duplicates and consolidated in your results.
- Crawl limits on your plan: Some Search Atlas plans include a maximum number of pages per crawl. If your site exceeds this limit, upgrade your plan or prioritise the most important sections of your sitemap.
- Sitemap URL not reachable: Confirm your sitemap returns a 200 HTTP status and is not behind a login, password protection, or firewall that blocks crawlers.
🔑 Quick Checklist
- Sitemap URL is accessible at a public address (no login required).
- Sitemap contains all pages you want audited — count the
<url>entries. - No unintended noindex tags or robots.txt blocks are in place.
- Correct sitemap URL is entered in your Search Atlas project settings.
- A fresh crawl has been started after any sitemap changes.
💬 Need More Help?
If you need further assistance, open the chat widget in the bottom-right corner of the platform and type human teammate to be connected with a member of our team.