🔍 Overview

This article covers two common questions from new agency users: why a WordPress site may not crawl after connecting, and how to get the most out of Search Atlas when onboarding new clients. Follow the steps below to resolve crawl issues and explore the platform's core capabilities.

⚙️ Why Your WordPress Site Isn't Crawling

After connecting WordPress to Search Atlas, a crawl may fail to start or return no data. This is almost always caused by one of the following:

• Plugin not activated: The Search Atlas WordPress plugin must be installed and activated on your site. Without it, the platform cannot communicate with your WordPress installation.
• Incorrect site URL: The URL entered during connection must exactly match your WordPress site address, including whether it uses www or not, and https vs http.
• Firewall or security plugin blocking crawlers: Plugins such as Wordfence, Cloudflare firewall rules, or password-protected staging environments can block the Search Atlas crawler. Temporarily whitelist the crawler IP or disable bot-blocking rules during the crawl.
• robots.txt disallowing crawl: Check your robots.txt file to ensure it does not block all bots or the Search Atlas user agent.
• Site is not publicly accessible: If your WordPress site is in maintenance mode or behind a login wall, the crawler cannot access it.

🔧 Steps to Fix a WordPress Crawl

1. Log in to your WordPress admin dashboard and confirm the Search Atlas plugin is installed and active under Plugins → Installed Plugins.
2. In Search Atlas, navigate to your site settings and verify the connected URL matches your live WordPress address exactly.
3. Check your robots.txt file by visiting yourdomain.com/robots.txt. Make sure no Disallow: / rule is blocking all bots.
4. If you use a security plugin or Cloudflare, temporarily pause bot-protection rules and attempt the crawl again.
5. Ensure the site is publicly accessible — disable maintenance mode and remove any login requirements on the front end.
6. Return to Search Atlas, open your project, and trigger a new crawl manually.

If the crawl still does not complete after following these steps, if you need further assistance, open the chat widget in the bottom-right corner of the platform and type human teammate to be connected with a member of our team.

🚀 How Agencies Onboard New Clients in Search Atlas

Search Atlas is built with agencies in mind. Here is how to structure your workflow when bringing on a new client:

1. Create a new project: Each client gets their own project inside Search Atlas. Navigate to your dashboard (Home) and select Create Project, then enter the client's domain.
2. Connect their WordPress site (if applicable): Install the Search Atlas plugin on the client's WordPress site and connect it to the project following the steps above.
3. Run a site audit: Trigger a crawl to identify technical SEO issues such as broken links, missing meta tags, slow pages, and crawl errors. Share the audit report with your client as a baseline.
4. Set up keyword tracking: Add the client's target keywords to the Rank Tracker to monitor positions over time and measure the impact of your work.
5. Use OTTO SEO for automation: Navigate to Left sidebar → OTTO SEO → All Sites (SEO Automation) (URL: /seo-automation-v3) to activate OTTO, Search Atlas's AI-powered SEO automation. OTTO can automatically fix on-page issues, deploy schema, and optimise content at scale — saving agencies significant time across multiple clients.

🔗 Search Atlas Backlink Capabilities for Agencies

Search Atlas includes a comprehensive backlink intelligence suite that agencies can use to analyse, monitor, and build links for clients:

• Backlink Analyzer: Enter any domain to see its full backlink profile, including referring domains, anchor text distribution, domain authority, and link types (dofollow vs nofollow).
• Competitor backlink gap: Compare your client's backlink profile against competitors to identify link-building opportunities they are missing.
• Link monitoring: Track newly acquired and lost backlinks over time so you can demonstrate link-building ROI to clients.
• Prospecting data: Use the platform's database to find high-authority sites in your client's niche for outreach campaigns.

These tools are available directly within each client project, making it easy to generate white-label-ready reports and insights for your agency deliverables.

💡 Agency Best Practices

• Use separate projects for each client to keep data, crawls, and reports isolated and organised.
• Schedule regular crawls so you catch technical regressions before they affect rankings.
• Combine the backlink analyzer with keyword tracking to build a full-picture SEO strategy for each client.
• Use OTTO SEO to handle repeatable optimisation tasks so your team can focus on strategy and client communication.

If you need further assistance, open the chat widget in the bottom-right corner of the platform and type human teammate to be connected with a member of our team.