πŸ› οΈ Stop OTTO Crawling Noindex URLs

Camilo Aponte

Camilo Aponte

Last updated on Nov 5, 2026

🚫 Why noindex URLs appear in OTTO

OTTO can discover more URLs than the pages you want optimized. A crawl may follow internal links, read your sitemap, or reuse URLs from an earlier scan. Pages marked noindex can still be found even though you do not want search engines to index them or OTTO to change them.

Deleting or ignoring a recommendation removes that task, but it does not stop OTTO from finding the same URL again. Exclude the URL, then run a new crawl, so it is not rediscovered.

🎯 Limit OTTO to your intended pages

Set the exclusion rules before you recrawl. Use them to keep only the URLs you want optimized, such as your intended pages.

  1. Open the OTTO project in Search Atlas, then open Settings and go to Crawl Parameters.
  2. Add an Exclude rule for paths you do not want optimized, such as noindex, thank-you, filter, tag, and other utility URLs. Use an Include rule only when you need an exception to an exclusion.
  3. Add or confirm the exact URLs you provided so the allowed set matches your intended pages.
  4. Exclude the unwanted URLs if the project has an exclude or ignore option. Excluding a URL is stronger than only deleting its recommendation.
  5. Check that important pages are still indexable. A page with a noindex tag can still be crawled and can still receive OTTO recommendations, so exclude it if you do not want it optimized.

Keep the allowed list tight. If OTTO can crawl the whole site again, noindex URLs may return in the next task list.

🧹 Clear old technical recommendations

Remove recommendations created from the wider crawl at any time. Save the exclusion rules before the next crawl so those URLs are not found again.

  • Open the OTTO project and select the relevant issue type in the task list.
  • Use Search by Page URL, or Filter by URL, to find the recommendations for pages outside your intended set, then select those rows.
  • Delete or ignore the selected recommendations. Ignore is the stronger task action because that issue type is not rebuilt for the page on the next crawl, but neither action changes which URLs the next crawl can discover.
  • Do not approve or deploy changes for excluded or noindex pages.

Clearing the list removes old work items. It does not by itself change what the next crawl is allowed to discover. Always update the included and excluded URLs first.

πŸ”„ Rescan using the corrected URL set

  1. Save the included and excluded URL settings.
  2. After saving the exclusion rules, open the project’s Site Audit and start Recrawl, or Recrawl Site, and wait for it to finish.
  3. Wait until the scan finishes before reviewing recommendations. A scan still in progress can show a mix of old and new results.
  4. Open the new task list and check that recommendations refer only to your intended pages.
  5. If any noindex URL remains, exclude it and run one more scan rather than editing that page through OTTO.

The new scan should rebuild recommendations from the pages still in scope. Tasks for removed URLs should not return when those URLs stay excluded.

βœ… Confirm the result

  • Compare the recommended URLs with your intended page list.
  • Search the task list for a few known noindex URLs. They should not have open optimization tasks.
  • Check that canonical, indexable pages you want to improve still have their recommendations.
  • Approve only the tasks that belong to the final URL set.

If the same unwanted URLs come back, check for a sitemap entry, an internal link, or a saved include rule that adds them on every crawl. Remove that source, keep the URL excluded, and rescan.

πŸ’¬ When to contact support

If you cannot clear the task list, the scan still includes noindex URLs after they are excluded, or you need help limiting OTTO to a specific URL set, contact support from inside the platform. If you need further assistance, open the chat widget in the bottom-right corner of the platform and send your message. A member of our support team will reply in the chat.

Was this article helpful?