🔍 Overview

After installing OTTO on your website, the crawler should systematically visit and index all pages within your connected domain. If OTTO reports only 1 page crawled after installation, your server or security layer is almost certainly blocking the Search Atlas crawler before it can follow internal links. This article walks you through diagnosing the block and whitelisting the crawler on the most common platforms.

⚠️ Symptoms That Confirm Crawler Blocking

• OTTO shows 1 page crawled and the count does not increase over time.
• The crawl appears to complete unusually fast (within seconds).
• Internal links on your site are not discovered or are shown as uncrawled.
• Your server logs show repeated 403, 429, or 503 responses from the Search Atlas crawler user-agent or IP range.
• You have recently added or updated a firewall, WAF, CDN, or bot-protection rule.

If you see one or more of these signs, proceed to the diagnosis steps below.

🛠️ Step 1 — Identify What Is Blocking the Crawler

Work through this quick diagnosis tree to pinpoint the source of the block before attempting a fix.

1. Check your robots.txt file. Visit yourdomain.com/robots.txt and look for any Disallow rules that could apply to all bots or to the Search Atlas user-agent specifically. If a broad Disallow: / rule exists, the crawler cannot access any page beyond the entry point.
2. Check for a WAF or bot-protection service. If your site uses Cloudflare, Sucuri, Wordfence, AWS WAF, Imperva, or a similar tool, its bot-blocking rules may be treating the Search Atlas crawler as a threat and returning a block page after the first request.
3. Check your hosting control panel. Some managed hosts (WP Engine, Kinsta, SiteGround, etc.) include built-in bot filtering that can block crawlers not on their approved list.
4. Check for rate-limiting rules. A rate limiter may allow the first request (the home page) and then drop all subsequent requests from the same IP, causing OTTO to stall at exactly 1 page.

✅ Step 2 — Whitelist the Search Atlas Crawler

Once you know where the block is coming from, use the relevant instructions below to allow the crawler through.

Search Atlas Crawler IP Addresses

Add the following IP addresses to your allowlist in whichever platform is blocking the crawler. Always refer to your OTTO dashboard or contact our team for the most current IP list, as addresses may be updated periodically.

• Navigate to OTTO SEO in the left sidebar, then open SEO Automation (URL: /seo-automation-v3).
• Open your connected website and select OTTO Settings.
• Scroll to the Crawler IP Addresses section to copy the current IP list directly from the platform.

Cloudflare

1. Log in to your Cloudflare dashboard and select the affected domain.
2. Go to Security → WAF → Tools.
3. Under IP Access Rules, enter each Search Atlas IP address and set the action to Allow.
4. Optionally, create a WAF Custom Rule that matches the Search Atlas user-agent string and sets the action to Skip all security rules.

Wordfence (WordPress)

1. In your WordPress admin, go to Wordfence → Firewall → Blocking.
2. Select Whitelist IPs that bypass all rules.
3. Add each Search Atlas crawler IP address and save.

Generic WAF or Hosting Firewall

1. Locate the IP allowlist or whitelist section in your WAF or hosting control panel.
2. Add each Search Atlas crawler IP as a trusted source.
3. If your platform supports user-agent rules, also add the Search Atlas crawler user-agent string available in your OTTO Settings panel.

robots.txt

1. Open your robots.txt file for editing.
2. Add a block explicitly allowing the Search Atlas crawler, using the exact user-agent string shown in your OTTO Settings panel, followed by Allow: /.
3. Make sure no subsequent Disallow rule overrides this permission.
4. Save and re-publish the file, then allow up to 15 minutes for the change to propagate.

🔄 Step 3 — Restart the OTTO Crawl

1. Go to OTTO SEO in the left sidebar, then open Site Audit → Overview (Website Overview).
2. Select your website to view the Website Overview page.
3. Click Recrawl Site to trigger a fresh crawl now that the IP addresses are whitelisted.

⏱️ Expected Crawl Completion Timelines

Crawl completion time varies with site size and how quickly the whitelisting change propagates — smaller sites generally complete faster than very large sites, and OTTO may continue processing large sites in the background after the initial pass. If a crawl seems to be taking unusually long relative to your site's size, contact support so we can check on its progress.

You can monitor real-time crawl progress from the OTTO SEO V3 dashboard. The page count will increment visibly once the block is resolved.

🚫 Common Mistakes to Avoid

• Whitelisting the IP in only one layer when multiple security tools are active — check every layer (CDN, WAF, host firewall, plugin).
• Forgetting to save or publish the whitelist changes before restarting the crawl.
• Confusing the Search Atlas crawler with Google bot and applying rules to the wrong user-agent.
• Not restarting the crawl after whitelisting — OTTO will not automatically retry until a new crawl is initiated.

💬 Still Need Help?

If you need further assistance, open the chat widget in the bottom-right corner of the platform and type human teammate to be connected with a member of our team.