🔍 Why OTTO Tracks Pages Even With Noindex Tags
Noindex tags and robots.txt directives tell search engines not to index or crawl a page — they do not automatically remove those URLs from OTTO's internal tracking list. When OTTO first crawls your site, it discovers and logs every reachable URL, including paginated URLs such as /page/2, /page/3, and query-string variants like ?paged=2. Those URLs remain in OTTO's database until they are explicitly excluded or removed, regardless of any noindex or robots.txt changes made after the initial crawl.
The key distinction: noindex controls what Google indexes. OTTO's tracked page count reflects every URL that has been discovered and queued for monitoring — these are two separate systems that do not automatically sync.
⚙️ How Crawl Limits and Robots.txt Interact With OTTO
Lowering the crawl limit in your project settings affects how many new pages OTTO discovers on future crawls — it does not retroactively remove URLs already added to your tracked list. Likewise, adding a robots.txt block prevents OTTO from re-crawling those paths on future runs, but does not instantly delete them from your current page count.
This is why you may see a high tracked page count (for example, 922 pages) even after applying all three changes — noindex tags, robots.txt blocks, and a lower crawl limit. The count only decreases once already-tracked URLs are actively excluded from OTTO's page list.
📋 How to Exclude Paginated URLs From OTTO
Follow these steps to remove paginated and noindexed URLs from OTTO's tracking and bring your page count down to your target.
- Go to Left sidebar → OTTO SEO → All Sites and click on your project card to open the project.
- Navigate to Left sidebar → OTTO SEO → Site Audit → Page Explorer. This screen lists every URL currently tracked by OTTO for your project.
- Use the search bar or URL filter to isolate paginated URLs — for example, filter by patterns containing /page/, ?paged=, or whichever pagination format your site uses.
- Select all paginated URLs you want to remove. Use the bulk-select checkbox at the top of the list to select all filtered results at once.
- Click Exclude or Remove from tracking to dequeue those URLs. OTTO will stop monitoring them, and they will no longer count toward your tracked page total.
- Confirm the action when prompted. Your tracked page count will update to reflect the removed URLs.
🔄 Preventing Paginated URLs From Being Re-Added
After removing the unwanted URLs, take these additional steps to stop them from reappearing in the next crawl cycle.
- Keep your robots.txt block in place — this signals OTTO not to re-crawl those URL patterns in future runs.
- Verify noindex tags are server-side rendered — go to Left sidebar → OTTO SEO → Site Audit → Crawl Monitoring to review which pages OTTO detected as noindexed. Tags rendered only via client-side JavaScript may not be read correctly during crawl; ensure they appear in the raw HTML response.
- Add a crawl exclusion rule — inside your project (accessible via Left sidebar → OTTO SEO → All Sites → [your project card]), look for URL exclusion or crawl rule settings and add a pattern to permanently block paginated paths, for example /page/* or ?paged=*. This prevents newly discovered paginated URLs from being added to your list in the future.
- Confirm your crawl limit is still set correctly — your lowered crawl limit reinforces the exclusion rules by capping how many pages OTTO queues per crawl cycle.
✅ Verifying Your Updated Page Count
After completing the steps above, go to Left sidebar → OTTO SEO → Overview to view your refreshed tracked page total. Allow up to one full crawl cycle for the count to update completely. If the count is still higher than expected, return to Left sidebar → OTTO SEO → Site Audit → Page Explorer and check for any remaining paginated or noindexed URLs that were not captured in your initial filter — for example, URLs with different pagination patterns or query strings.
If you need further assistance, open the chat widget in the bottom-right corner of the platform and type human teammate to be connected with a member of our team.