# **🔍 Overview**

If your site has a large number of pages, you may notice that OTTO SEO has crawled and indexed fewer pages than your total page count. This is expected behaviour. OTTO uses a smart crawling system designed to prioritise the pages most likely to impact your SEO performance, while keeping resource usage efficient for your site.

# **📊 Why OTTO Indexes a Subset of Your Pages**

OTTO does not crawl every page on your site in a single pass. Instead, it works through your pages progressively, applying the following logic:

- **Priority-based crawling:** OTTO focuses first on your highest-value pages — typically those already receiving organic traffic, linked from your sitemap, or flagged as important in your site architecture.
- **Crawl budget allocation:** Each connected site is assigned a crawl budget based on your Search Atlas plan. This budget controls how many pages OTTO processes per crawl cycle.
- **Incremental indexing:** Pages are added to OTTO's index over multiple crawl cycles. A site with 65,000 pages will not be fully indexed after a single crawl — this is normal and by design.
- **Duplicate and low-value page filtering:** OTTO automatically excludes near-duplicate pages, thin content pages, and URLs blocked by your robots.txt or noindex directives, which can reduce the indexed count compared to your total page count.

In the example of a 65,000-page site showing 16,000 indexed pages, this reflects the progress of the current crawl cycle combined with priority filtering. The indexed count will grow with each subsequent crawl.

# **🕐 When Does OTTO Crawl Your Site?**

OTTO is designed to crawl during **off-peak hours** to minimise any impact on your site's performance and server load. Here is what you need to know about the crawl schedule:

- **Off-peak timing:** Crawls are scheduled to run during low-traffic periods, typically overnight based on the time zone associated with your Search Atlas account.
- **Automatic scheduling:** You do not need to manually trigger crawls. OTTO manages the schedule automatically once it is activated for your site.
- **Recurring cycles:** OTTO re-crawls your site on a regular basis to detect new pages, content changes, and resolve previously identified issues. The frequency depends on your plan tier.
- **On-demand crawl triggers:** If you publish a large batch of new pages or make significant structural changes, you can manually request a fresh crawl from within the OTTO dashboard to accelerate indexing.

# **🛠️ How to Check Crawl Status and Progress**

1. Log in to Search Atlas and navigate to **OTTO SEO → All Sites (SEO Automation)** in the left sidebar, or go directly to **/seo-automation-v3**.
2. Select the site you want to review from the site selector at the top of the page.
3. Review the crawl summary panel, which shows the total pages discovered, pages indexed, and the timestamp of the last completed crawl.
4. Check the **Crawl Progress** indicator to see how far through the current cycle OTTO has progressed.
5. If you have recently added new pages and want OTTO to pick them up sooner, use the **Trigger Crawl** option if available on your plan.

# **💡 Tips to Improve Crawl Coverage**

- **Submit an updated XML sitemap:** Ensure your sitemap is connected to OTTO and reflects your full page inventory. OTTO uses the sitemap as a primary discovery source.
- **Audit your robots.txt:** Confirm that important pages are not unintentionally blocked from crawling.
- **Remove noindex tags from pages you want indexed:** OTTO respects noindex directives and will exclude those pages from its count.
- **Reduce redirect chains:** Long redirect chains slow down crawling and can cause pages to be skipped within a given cycle.
- **Upgrade your plan for a higher crawl budget:** If your site grows significantly, a higher-tier plan increases the number of pages OTTO can process per cycle.

# **❓ Frequently Asked Questions**

**Will OTTO eventually index all 65,000 pages?** Yes — over multiple crawl cycles OTTO will work through your full page inventory, subject to your plan's crawl budget. Pages that are accessible, indexable, and not filtered as low-value will be added incrementally.

**Does crawling affect my site speed?** OTTO's off-peak scheduling and rate-limited crawling are specifically designed to avoid impacting your site's performance or server response times.

**Why did the indexed page count drop?** A drop in indexed pages usually means OTTO detected that some pages became noindexed, were removed, returned errors (such as 404s), or were filtered out as duplicates during the latest crawl cycle.

# **🙋 Still Need Help?**

If you need further assistance, open the chat widget in the bottom-right corner of the platform and type **human teammate** to be connected with a member of our team.