🔍 Overview

After installing OTTO SEO, some customers notice that the crawler only indexes a single page — typically the homepage — instead of crawling the entire website. This usually happens because a security plugin, firewall, or server-level rule is blocking the Search Atlas crawler bot after its first request. This article walks you through how to identify the cause and whitelist the crawler correctly so OTTO can index your full site.

⚙️ How the OTTO Crawler Works

When OTTO is activated for your website, the Search Atlas crawler visits your pages to gather SEO data and apply optimizations. If anything on your server interprets repeated bot requests as suspicious activity, it may block the crawler after the first visit — leaving OTTO with only partial data to work with.

Common causes of this issue include:

• Security plugins (e.g., Wordfence, Sucuri, iThemes Security) blocking unrecognized bots
• Cloudflare or CDN firewall rules that rate-limit or challenge bot traffic
• Hosting-level bot protection that restricts crawlers automatically
• robots.txt rules that unintentionally disallow the Search Atlas crawler

🛠️ Step 1 — Whitelist the Search Atlas Crawler User-Agent

The Search Atlas crawler identifies itself with the following user-agent string:

SearchAtlas/1.0

You need to add this user-agent to the allowlist in any tool that filters bot traffic on your site. Follow the steps below based on your setup.

🔒 Step 2 — Whitelist in Your Security Plugin

If you are using a WordPress security plugin, here is how to allow the crawler:

1. Log in to your WordPress dashboard.
2. Navigate to your security plugin settings (e.g., Wordfence → Firewall → Allowlisted IPs or Bots).
3. Locate the option to whitelist a bot or user-agent.
4. Add SearchAtlas/1.0 as a trusted user-agent.
5. Save your changes.

If your plugin does not support user-agent whitelisting, check whether it allows you to whitelist by IP address. Contact our support team via the chat widget (see below) to request the current list of Search Atlas crawler IP addresses.

☁️ Step 3 — Whitelist in Cloudflare or Your CDN

If your site uses Cloudflare or another CDN with bot management, follow these steps:

1. Log in to your Cloudflare dashboard and select your domain.
2. Go to Security → WAF → Custom Rules (or Firewall Rules, depending on your plan).
3. Create a new rule that allows traffic where the user-agent contains SearchAtlas.
4. Set the action to Allow and save the rule.
5. If you use Cloudflare Bot Management or Super Bot Fight Mode, ensure verified bots or the Search Atlas user-agent are not being challenged or blocked.

📄 Step 4 — Check Your robots.txt File

Your robots.txt file may be blocking the Search Atlas crawler. To check:

1. Open a browser and go to yourdomain.com/robots.txt.
2. Look for any Disallow rules under a wildcard agent (User-agent: *) that might block large portions of your site.
3. If needed, add a specific entry to explicitly allow the Search Atlas crawler:

User-agent: SearchAtlas/1.0
Allow: /

Add these lines above any wildcard disallow rules so they take priority. Save and re-upload the file to your server root if you edit it manually.

🚀 Step 5 — Re-Trigger the OTTO Crawl

Once you have completed whitelisting, you need to prompt OTTO to re-crawl your site:

1. In Search Atlas, go to OTTO SEO in the left sidebar, then open SEO Automation.
2. Select the website where OTTO is installed.
3. Locate the crawl or re-crawl option and initiate a new crawl.
4. Wait for the crawl to complete — this may take a few minutes depending on your site size.
5. Review the crawled pages count to confirm all pages are now being discovered.

✅ Confirm the Issue Is Resolved

After the re-crawl, check the following to confirm everything is working:

• The total pages crawled matches the expected number of pages on your site.
• OTTO is surfacing SEO recommendations across multiple pages, not just the homepage.
• No new error messages appear in the OTTO dashboard related to crawl access.

If the crawled page count is still lower than expected, double-check that all layers of protection on your site — including hosting-level firewalls and CDN rules — have been updated.

💬 Still Need Help?

If you need further assistance, open the chat widget in the bottom-right corner of the platform and type human teammate to be connected with a member of our team.