🚫 Exclude Unwanted URLs from OTTO Scans

Camilo Aponte

Camilo Aponte

Last updated on Nov 6, 2026

🚫 Why unwanted URLs appear in OTTO

OTTO uses discovered and crawled pages to build recommendations. Accessible archives, parameter URLs, and other unwanted paths can enter that set. Crawl exclusions control which URLs participate; a recrawl refreshes crawl data but does not replace every existing recommendation wholesale.

You do not need to delete each recommendation by hand. Exclude the URLs you do not want, then rescan. After the new scan finishes, the task list should reflect only the included pages.

πŸ“‹ Before you change the scan

  • List the real pages you want scanned, such as your 136 live URLs.
  • List the URLs or path patterns you want left out, such as /tag/, /cart/, or query-string pages.
  • Confirm those pages are not required for navigation or important internal links.
  • Wait until any scan already in progress has finished before you start a new one.

βš™οΈ How to exclude URLs and rescan

  1. Open the site in Search Atlas and go to OTTO.
  2. Open the scan or crawl settings for that site.
  3. Under Crawl Parameters, find URL Exclusion conditions and click + Add condition. Choose Exclude, select Contains or Start With, and enter the unwanted URL path or matching path text, such as /tag/. Add additional conditions as needed. (customer-frontend/src/modules/otto/shared/components/settings-modal/ottoSettings.tsx:839–845,1197–1256.)
  4. Use Include only when you need an exception to an Exclude condition. OTTO requires an Exclude condition before enabling Include; this is not a separate list of pages to review. (customer-frontend/src/modules/otto/shared/components/settings-modal/ottoSettings.tsx:1217–1224.)
  5. Click Update Settings to save the exclusion conditions. Then open the linked Site Audit and choose Recrawl or Recrawl Site. (customer-frontend/src/modules/otto/shared/components/settings-modal/ottoSettings.tsx:1287–1300; src/modules/site-audit/v2/shared/recrawl-button/index.tsx:30.)
  6. When the scan completes, review the task list. Recommendations for excluded URLs should no longer appear.

Excluding a URL stops it from driving new OTTO tasks. It does not delete the page from your website, change its content, or remove it from Google by itself.

πŸ” Can you purge the task list?

You do not need a delete-all action to exclude unwanted pages from active recommendations. Save the exclusion conditions and recrawl the linked Site Audit. OTTO marks matching existing URLs as excluded and filters them from active results; the recrawl updates existing page data rather than deleting and rebuilding every recommendation. (content-assistant/app/ca/models/otto\_project.py:7913–7953; app/ca/pipelines/otto\_v2.py:7060–7064.)

If old tasks for excluded URLs are still visible after the new scan shows as complete, refresh the OTTO view. If they remain, use the chat widget so support can check that scan.

πŸ” Does OTTO follow noindex rules?

No. A noindex tag is not a reliable way to keep a URL out of an OTTO scan. Noindex tells search engines not to index a page. It does not always stop a site audit from crawling that page or creating technical recommendations for it.

  • Use OTTO exclusions when you want a page left out of scans and the task list.
  • Use noindex when you want search engines to drop a page from their index.
  • A robots.txt disallow blocks crawling only when Respect robots.txt is enabled. It does not immediately erase previously stored OTTO recommendations. Use URL Exclusion conditions for explicit OTTO scope control, then recrawl and review the results. (content-assistant/app/ca/models/site\_audit.py:706–717; app/ca/models/otto\_project.py:7913–7943.)

Pages you want audited should remain accessible to the crawler. They may use noindex when that matches your indexing policy. For pages you want excluded from OTTO, configure URL Exclusion conditions even if those pages already use noindex. (content-assistant/app/ca/models/otto\_url.py:118–131.)

βœ… After the new scan

  • Check that the page count is close to your real set, not the old larger set.
  • Open a few excluded URLs in the results and confirm they no longer have open tasks.
  • Spot-check important live pages to confirm their recommendations were regenerated.
  • Keep the exclusion list updated when you add new template, filter, or parameter URLs.

πŸ’¬ Need help with a scan

If you need further assistance, open the chat widget in the bottom-right corner of the platform and send your message. A member of our support team will reply in the chat.

Was this article helpful?