🗺️ How Search Atlas Crawls Your Website
Search Atlas uses your sitemap to discover and crawl pages on your website. Unlike some third-party tools that actively spider every link they find, Search Atlas relies primarily on a valid, well-structured XML sitemap to identify which pages to audit and index. If your sitemap is incomplete, misconfigured, or missing, Search Atlas may not crawl all of your pages — even if another tool successfully did so using a different discovery method.
⚠️ Common Reasons Not All Pages Are Crawled
- Incomplete sitemap: Your sitemap does not list all 500+ pages. Pages left out of the sitemap will not be discovered by Search Atlas.
- Sitemap not submitted or linked: Your sitemap URL has not been provided to Search Atlas, or it is not referenced in your robots.txt file.
- Multiple sitemaps not included: Large sites often use a sitemap index file that references multiple child sitemaps. If only one child sitemap is submitted, other pages will be missed.
- Sitemap contains errors: Malformed XML, incorrect URLs, or pages marked with a noindex directive may cause Search Atlas to skip those entries.
- Crawl limit not yet reached: Depending on your plan, there may be a page crawl limit. If your site exceeds this limit, not all pages will be processed in a single crawl.
- Robots.txt is blocking crawlers: Directives in your robots.txt file may be preventing Search Atlas from accessing certain sections of your site.
✅ How to Resolve Incomplete Crawls
- Verify your sitemap is complete. Open your sitemap (typically found at yourdomain.com/sitemap.xml) and confirm every page you want crawled is listed. Use a sitemap validation tool to check for formatting errors.
- Check for a sitemap index file. If your site has more than a few hundred pages, it likely uses a sitemap index. Make sure the index file references all child sitemaps and that you submit the index URL — not just one child sitemap — to Search Atlas.
- Submit the correct sitemap URL to Search Atlas. Navigate to your project settings inside the platform and confirm the sitemap URL entered matches the exact location of your sitemap or sitemap index file.
- Review your robots.txt file. Visit yourdomain.com/robots.txt and check that no Disallow rules are blocking the pages you want crawled. Also confirm your sitemap URL is referenced with a Sitemap: directive.
- Remove noindex tags from pages you want indexed. If a page has a noindex meta tag or X-Robots-Tag header, Search Atlas will respect that directive and exclude the page. Remove the tag if you want the page to be crawled and indexed.
- Re-trigger a crawl. After correcting your sitemap or settings, initiate a new crawl from within your Search Atlas project to pick up the updated page list.
📊 Understanding Plan-Based Crawl Limits
Search Atlas crawl limits vary by subscription plan. If your website has more pages than your current plan allows in a single crawl, only a portion of your site will be processed. To crawl larger sites in full, consider upgrading your plan or prioritising the most important pages in your sitemap by placing them at the top of the file. Contact our team via the chat widget for guidance on which plan suits your site size.
🛠️ Using On-Page Audit to Review Crawled Pages
Once a crawl is complete, you can review which pages were successfully indexed and identify any issues by navigating to Left sidebar → Content → On-Page Audit. This view shows all crawled URLs along with any errors, warnings, or notices that may be affecting coverage. Use this report to cross-reference against your sitemap and identify gaps in crawl coverage.
💡 Best Practices for Full Site Crawling
- Keep your sitemap up to date whenever new pages are published or old pages are removed.
- Use a sitemap index file for sites with more than 1,000 pages.
- Set the lastmod date on each sitemap entry so Search Atlas can prioritise recently updated pages.
- Avoid including pages in your sitemap that return a 301 redirect, 404 error, or noindex directive — these waste crawl budget.
- Validate your sitemap regularly at yourdomain.com/sitemap.xml to catch formatting issues early.
🙋 Still Seeing Missing Pages?
If you have followed the steps above and Search Atlas is still not crawling all of your pages, there may be a configuration issue specific to your site or account. If you need further assistance, open the chat widget in the bottom-right corner of the platform and type human teammate to be connected with a member of our team.