🧩 Why This Happens
Site Audit relies on being able to discover and access every page on your site. When pages are missing from the crawl, it usually means one or more common barriers are preventing the crawler from reaching them. This is especially common on WordPress sites due to plugin configurations, crawl limits, or technical settings that block or restrict access.
✅ Step 1: Check Your Crawl Settings
Before investigating deeper, confirm your project is configured correctly inside Search Atlas.
- Navigate to OTTO SEO → Site Audit from the left sidebar.
- Select your site and open Settings or Edit Project.
- Verify the Start URL is set to your root domain (e.g., https://yourdomain.com) and not a subfolder or inner page.
- Check the crawl limit. If your site has hundreds or thousands of pages, make sure the limit is set high enough to cover all of them.
- Ensure crawl subdomains is enabled if you have content hosted on subdomains.
🚫 Step 2: Review Your robots.txt File
A misconfigured robots.txt file is one of the most common reasons pages are skipped during a crawl. If your robots.txt is blocking the Search Atlas crawler — or all crawlers — entire sections of your site will be invisible to Site Audit.
- Open a browser and go to https://yourdomain.com/robots.txt.
- Look for any Disallow rules that block User-agent: * or specific paths.
- If important pages or directories are disallowed, update your robots.txt to allow access to them.
- After updating, return to Site Audit and re-run the crawl.
🔌 Step 3: Disable or Configure WordPress Plugins
Several WordPress plugins can interfere with crawling without you realising it. The most frequent offenders include:
- Security plugins (e.g., Wordfence, iThemes Security) — these can block crawlers that resemble bots. Whitelist the Search Atlas crawler IP or temporarily disable bot blocking rules to test.
- Caching plugins (e.g., WP Rocket, W3 Total Cache) — some caching rules can cause redirect loops or serve incomplete pages to crawlers.
- Coming Soon / Maintenance mode plugins — if your site or any section is in maintenance mode, those pages will not be crawled.
- Membership or login-required plugins — pages behind a login wall cannot be crawled unless you configure authenticated crawling.
Test by temporarily deactivating suspected plugins, then re-running the Site Audit crawl to see if more pages are discovered.
🗺️ Step 4: Verify Your XML Sitemap
Site Audit can use your XML sitemap as a reference to find pages. If your sitemap is missing pages, outdated, or broken, the crawl may not discover all content.
- Go to https://yourdomain.com/sitemap.xml (or your custom sitemap URL).
- Confirm all important pages are listed.
- If you use a WordPress SEO plugin like Yoast SEO or Rank Math, check the sitemap settings to ensure all post types and taxonomies are included.
- Regenerate the sitemap if needed, then re-submit it in Site Audit settings.
🔗 Step 5: Fix Internal Linking Gaps
The Site Audit crawler discovers pages by following links. If some pages have no internal links pointing to them (orphan pages), the crawler cannot find them through link traversal alone.
- Add internal links from other pages to any orphaned content.
- Ensure your navigation menus, category pages, and footer links cover all key areas of your site.
- Use your XML sitemap alongside crawling to maximise page discovery.
🔄 Step 6: Re-Run the Crawl
After making any of the changes above, always trigger a fresh crawl so Site Audit can process the updated site.
- Go to OTTO SEO → Site Audit → Overview (Website Overview) in the left sidebar.
- Select your project and click Recrawl Site.
- Wait for the crawl to complete, then review the Page Explorer report to confirm all expected pages are now listed.
- Check the Issues report to view and action any errors that were previously missed.
💬 Still Need Help?
If you have followed all the steps above and Site Audit is still not crawling all pages on your WordPress site, If you need further assistance, open the chat widget in the bottom-right corner of the platform and type human teammate to be connected with a member of our team.