## **🔍 What is crawled page inventory?**

Crawled page inventory is the collection of URLs that Search Atlas discovers and processes during a site crawl. It helps the platform understand your site structure, identify linking opportunities, and evaluate how pages connect to one another.

This inventory is different from your website’s total number of pages, the number of pages indexed by search engines, or the number of URLs listed in your CMS.

## **🕷️ How are pages discovered?**

Search Atlas typically discovers pages using the crawlable links and site information available to the crawler, including:

- Internal links found in page content, navigation, headers, footers, and other crawlable areas
- XML sitemap URLs that are available to the crawler
- URLs found while following links from previously discovered pages
- Configured crawl settings, such as the starting URL, allowed pages, and crawl limits

A URL may appear in a sitemap but remain unavailable for analysis if it cannot be fetched, is blocked, redirects unexpectedly, requires authentication, or falls outside the configured crawl scope.

## **📊 What the inventory represents**

The inventory represents the pages Search Atlas could discover and successfully process during the relevant crawl. It may include pages with different statuses, such as successful pages, redirects, errors, or blocked URLs, depending on the report or feature being viewed.

The inventory can change after a new crawl. Changes to internal links, sitemaps, robots.txt, redirects, canonical tags, site availability, or crawl settings may increase or reduce the number of available pages.

## **🔗 How internal linking uses crawl data**

Internal linking features use the crawled inventory to identify relevant source and destination pages. Recommendations depend on the pages discovered, the content available for analysis, and the links already detected during the crawl.

If a page is missing from the inventory, it generally cannot be used reliably as a source or destination for internal linking recommendations until it is discovered and processed successfully.

## **🛠️ Why a page may be missing**

- The page is not linked from crawlable pages.
- The URL is missing from the XML sitemap.
- Robots.txt or page-level directives prevent crawling or indexing.
- The page returns an error, redirect, timeout, or server response that prevents processing.
- The page requires a login or other restricted access.
- The URL is outside the project scope or crawl limits.
- The page was added or changed after the last crawl.
- URL variations, parameters, or canonical settings cause the crawler to treat it differently.

## **✅ How to improve crawl coverage**

1. Confirm that the page returns a successful response when opened without authentication.
2. Add important pages to the XML sitemap.
3. Link to those pages from relevant, crawlable pages on the site.
4. Review robots.txt, noindex directives, canonical tags, and redirects.
5. Check the project’s crawl scope, limits, and starting URL.
6. Run a new crawl after making changes, then review the updated inventory.

## **💡 Important expectations**

Crawl inventory is a snapshot of what the crawler could discover and process at a specific time. It is not a real-time list of every page on the website and does not guarantee that a page is indexed by Google or another search engine.

For the most accurate internal linking recommendations, keep your sitemap current, maintain clear internal navigation, and recrawl after significant site changes.

## **💬 Need help reviewing an escalation?**

When contacting support, include the project name, affected URL, crawl date, and any relevant crawl or sitemap details. This helps the team compare the page with the crawl results and determine why it was included or excluded.

If you need further assistance, open the chat widget in the bottom-right corner of the platform and send your message. A member of our support team will reply in the chat.