🧭 Overview

OTTO, Search Atlas's OTTO SEO automation engine, crawls your site to discover and apply on-page optimizations. By default, OTTO will process all pages it can reach within your project. If you want to prevent OTTO from crawling or optimizing certain pages — such as Spanish-language pages, staging URLs, or low-priority sections — you do this through Crawl Settings inside Search Atlas, not through robots.txt or noindex tags.

This article explains how to set up OTTO-specific exclusions, clarifies whether OTTO changes are permanently written to your site, and covers how OTTO's reporting and integration features work.

⚠️ OTTO Exclusions vs. robots.txt — Key Difference

A common point of confusion is the difference between site-wide crawl rules and OTTO-specific exclusions.

• robots.txt / noindex tags — These are server-level directives that instruct all crawlers (Google, Bing, OTTO, etc.) how to behave. Editing robots.txt does not exclusively affect OTTO and is not the correct method for managing what OTTO processes.
• OTTO Crawl Settings — These are exclusion rules configured entirely within Search Atlas. They tell OTTO which URLs to skip during its crawl and optimization passes, without touching your site's server configuration or affecting other crawlers.

Always use OTTO Crawl Settings when you want to control what OTTO specifically crawls or optimizes.

🚫 How to Exclude Pages from OTTO Crawl

To exclude specific pages, directories, or URL patterns from OTTO's crawl, you will need to configure exclusion rules inside OTTO's Crawl Settings for your project. Because the exact navigation path to these settings may vary depending on your account setup, we recommend reaching out to our support team so they can guide you directly to the correct location and confirm the right exclusion pattern for your use case (for example, excluding all Spanish-language pages via a language subdirectory pattern).

When your exclusion rules are saved, OTTO will skip all matching URLs during its next crawl cycle. Any optimizations OTTO had already applied to those pages before the exclusion was added will remain in place unless you manually revert them.

🔄 Dynamic vs. Permanent Changes

A frequent question is whether OTTO's optimizations are written permanently to your site.

• Dynamic changes — Some OTTO optimizations are applied dynamically and are managed through Search Atlas. If you disconnect OTTO or remove a rule, those changes can be rolled back without a permanent edit to your CMS or server files.
• Permanent changes — Other optimizations (such as those pushed directly to your CMS) are written to your site and persist independently of OTTO. These require manual intervention to revert.

If you are unsure whether a specific OTTO action is dynamic or permanent, contact our support team before making changes.

📊 Reporting and Integrations

OTTO provides reporting on the optimizations it has applied across your project. You can review which pages have been processed, which changes are active, and track the impact of OTTO's work over time. OTTO also integrates with your CMS, allowing certain optimizations to be pushed directly to your site from within Search Atlas.

If you need further assistance, open the chat widget in the bottom-right corner of the platform and type human teammate to be connected with a member of our team.