🔍 Why OTTO Still Tracks Pages After Noindex or Robots.txt Changes
A common misconception is that adding noindex tags or robots.txt blocks will reduce the number of pages OTTO tracks. These directives tell search engines to ignore or skip certain pages — but they do not stop OTTO from counting a URL it has already successfully crawled.
OTTO tracks every URL it can reach during a crawl, regardless of whether those pages are marked noindex or blocked for search engines. The tracked-page count reflects OTTO's crawl scope, not your site's indexability in Google or Bing.
This means you can have a site with 922 tracked pages where hundreds of those pages carry noindex tags — OTTO will still count all 922 because it crawled them successfully.
📋 What Counts as a Tracked Page
OTTO adds a URL to its tracked-page count when all of the following are true:
- The URL was discovered during a crawl (via internal links, sitemaps, or pagination).
- The server returned a successful response (typically a 200 status code).
- The URL was not excluded through OTTO's own exclusion settings before the crawl ran.
Noindex meta tags, X-Robots-Tag headers, and robots.txt disallow rules are not evaluated by OTTO when deciding whether to count a page. They are signals for search engine crawlers only.
⚙️ How to Reduce Your Tracked Page Count
To bring your tracked-page count down, you need to reduce the number of URLs OTTO discovers and counts during its crawl. The key approaches are lowering the crawl limit and excluding specific URL patterns from OTTO's crawl scope — for example, paginated URLs, tag archives, or filtered product pages that do not need to be tracked.
These exclusions must be configured within OTTO's own crawl settings before the crawl runs. Applying noindex or robots.txt changes after pages have already been crawled will not retroactively remove them from the tracked count.
🔄 When to Contact Support
If you are unsure which crawl settings or exclusion options are available for your account, or if your tracked-page count does not decrease after making changes, please reach out to our support team via the chat icon in your dashboard. We can review your specific configuration and guide you through the correct steps to reach your target page count.