Crawling & Indexing: Guides

🔍 Forensic Auditing in Real Time: Clearing Out Cobwebs and Technical Debt

Modern websites accumulate technical debt quietly. Forgotten URLs, outdated test pages, low-value archives, and misconfigured redirects often live unnoticed inside a CMS, slowly draining crawl budget, internal link equity, and indexing efficiency. Traditional SEO audits surface these issues only after a crawl finishes, making it difficult to understand how and why they were discovered in the first place. Search Atlas provides real-time crawl monitoring and forensic page analysis, allowing teams to watch a crawl unfold live, identify hidden “cobwebs,” and trace their technical impact at the exact moment they are discovered. 🧠 Live Crawl Logs — See Your Site the Way Crawlers Do Search Atlas provides Live Crawl Logs, a real-time record of a site audit as it happens. During a crawl, the platform surfaces every discovered URL, HTTP status codes (200, 301, 404), and exact timestamps for each discovery. This live feed removes the black box from technical auditing. Instead of waiting for a crawl to finish, teams can immediately see which sections of the site are being explored, where crawl budget is being spent, and whether unexpected URLs are being indexed. This is especially valuable for large sites, enterprise environments, and long-lived WordPress installations. 🕸️ Identifying “Cobwebs” That Steal Equity One of the most powerful outcomes of live crawl visibility is the ability to surface forgotten or low-value pages that silently compete with your most important URLs. Common examples include old test URLs, default CMS pages, auto-generated date, tag, or archive pages, and thin internal landing pages that were never deprecated. These pages consume crawl budget, receive internal links, dilute link equity away from money pages, and increase indexing noise. By exposing these URLs in real time, Search Atlas allows teams to identify and act on technical debt before it compounds further. 🔬 Page Explorer — URL-Level Forensic Analysis Every URL surfaced in the live crawl log links directly to Page Explorer, where teams can perform deep, forensic analysis on individual pages. Page Explorer provides indexability status, HTTPS and protocol resolution, status code validation, download and response time metrics, sitemap inclusion checks, and internal and external link analysis. A key differentiator is incoming internal link visibility. Because Search Atlas performs a full crawl, it can identify exactly which internal pages link to a given URL, a level of insight that is impossible without crawling the entire site. This makes it easy to diagnose why low-value pages continue to receive authority and how equity flows through the site. When connected to first-party data sources, Page Explorer can also surface Search Console performance, click-through rates and positions, and analytics-based engagement and revenue signals, so teams can prioritize fixes based on technical impact and business value, not just crawl errors. ⚙️ JavaScript Rendering — Crawl Like Googlebot For modern, JavaScript-heavy sites and single-page applications, Search Atlas includes an optional JavaScript rendering toggle. When enabled, the crawler processes HTML, CSS, and JavaScript, so the audit reflects exactly what Googlebot sees, ensuring client-side rendered content, injected elements, and dynamic changes are fully evaluated. JS rendering should only be enabled when needed, as it increases crawl time. It is automatically enabled when required (for example, pixel-based implementations). Server-side rendered environments (WordPress plugin or Cloudflare integration) do not require JS rendering, since changes are already present in the initial HTML. 🔎 Why This Matters for Technical SEO and AI Search Real-time crawl logs and forensic page analysis are more than convenience features, they are foundational infrastructure for modern SEO. They allow teams to detect technical debt early, eliminate crawl waste, preserve internal link equity, improve indexing clarity, and reduce noise for search engines and AI systems. In an environment where both search engines and large language models favor clean, structured, and unambiguous sites, understanding how your site is crawled, as it happens, is no longer optional. Search Atlas turns site auditing from a static report into a live diagnostic system, giving teams the visibility they need to keep websites technically healthy at scale. If you need further assistance, open the chat widget in the bottom-right corner of the platform and type human teammate to be connected with a member of our team.

⏱️ How Long Does a Site Audit Take?

🔍 Overview When you run a Site Audit in Search Atlas, the tool crawls every page of your website to check for technical SEO issues. The time it takes depends on the size and complexity of your site. This article explains what to expect and how to get the most out of the process. ⏳ Typical Audit Durations Most Site Audits complete within the following timeframes: - Small sites (under 500 pages): Usually finishes in a few minutes. - Medium sites (500–5,000 pages): Typically takes 10–30 minutes. - Large sites (5,000+ pages): Can take 30 minutes to several hours depending on server response times and crawl settings. These are estimates. Actual duration may vary based on the factors listed below. ⚙️ What Affects Audit Speed Several factors can make an audit run faster or slower: - Total number of pages: The more pages your site has, the longer the crawl takes. - Server response time: If your server responds slowly to requests, the crawl slows down to match. - Crawl rate settings: A lower crawl rate reduces load on your server but increases audit time. - Redirects and complex URL structures: Sites with many redirects or deeply nested pages take longer to fully map. - JavaScript-heavy pages: Pages that rely heavily on JavaScript to render content may require additional processing time. 📊 How to Check Audit Progress You can monitor your audit in real time directly inside the platform: 1. In the left sidebar, go to Site Metrics → Site Explorer (Site Explorer). 2. Select the property you are auditing. 3. The audit status and progress indicator will be visible at the top of the page. You do not need to keep the page open for the audit to continue — it runs in the background and your results will be ready when you return. 💡 Tips While You Wait There is no need to sit and watch the progress bar. Here are a few productive things you can do while your audit runs: - Review existing keyword data in Keywords → Keyword Magic or Rank Tracker. - Check competitor gaps using Keywords → Keyword Gap. - Plan your next content piece in OTTO SEO → All Sites (SEO Automation) (left sidebar → OTTO SEO → All Sites). 🛠️ What to Do If an Audit Seems Stuck If your audit has been running for significantly longer than the estimates above and shows no progress, try the following steps: 1. Refresh the page to check whether the status has updated. 2. Confirm your site is publicly accessible and not blocking crawlers via robots.txt. 3. Check that your domain is correctly verified and active inside the platform. 4. If the issue persists, cancel the audit and re-run it. If you need further assistance, open the chat widget in the bottom-right corner of the platform and type human teammate to be connected with a member of our team.

📄 Pagination Variants vs. Duplicate URLs in Search Atlas

Overview If Search Atlas is showing multiple versions of your blog post URLs — for example, example.com/blog/post/, example.com/blog/post/?page=2, and example.com/blog/post/?page=3 — these are pagination variants, not duplicate pages created by Search Atlas. They are separate, search-engine-visible URLs that exist on your site and are being surfaced so you can manage them correctly. This article explains what pagination variants are, why they appear in your audit, and how to resolve any canonicalization issues. What Are Pagination Variants? Pagination variants are numbered URL versions of a single page. They are generated by your CMS or website platform when long-form content — such as a blog post, product listing, or article — is split across multiple pages. Common formats include: - example.com/blog/post/?page=2 - example.com/blog/post/2/ - example.com/blog/post/?p=2 Search engines crawl and index each of these URLs independently. Without correct canonical tags, search engines may treat them as competing versions of the same content, which can dilute your page authority and hurt rankings. Search Atlas does not create these URLs. It detects them during your site crawl and flags them so you can take action. How to Find Pagination Variant Issues in Your Audit Once your site crawl is complete, navigate to your audit project in Search Atlas and look for any issues related to duplicate or paginated URLs. The audit will surface grouped URL sets — showing the parent URL alongside its numbered variants — so you can review which pages are affected. If you are unsure where to locate these results within your specific audit view, our support team can guide you to the correct section of the interface. How to Fix Pagination Variants The correct fix depends on how your content is structured. The following approaches are commonly used to resolve pagination canonicalization issues. Option 1 — Add a canonical tag to each paginated URL Point the canonical tag on each paginated variant back to the main (first) page of the post. For example, example.com/blog/post/?page=2 should include a canonical tag referencing example.com/blog/post/. Add this inside the <head> section of each paginated page: - <link rel="canonical" href="https://example.com/blog/post/" /> Many CMS platforms allow you to set canonical URLs through SEO settings or plugins without editing code directly. Refer to your CMS documentation for the specific steps that apply to your platform. Option 2 — Disable pagination if it is not needed If your blog posts do not require pagination, consider disabling it in your CMS settings. This will prevent the additional paginated URLs from being generated and crawled. If you need further assistance, open the chat widget in the bottom-right corner of the platform and type human teammate to be connected with a member of our team.

⏱️ Site Audit Duration and Processing Time Explained

🔍 Overview Running a site audit in Search Atlas is one of the most thorough ways to identify technical SEO issues across your website. Because the audit crawls every page and analyses dozens of data points, it naturally takes some time to complete. This article explains what a normal audit duration looks like and what factors influence processing time. ⏳ How Long Does a Site Audit Take? Audit duration varies depending on the size and complexity of your website. Here are general benchmarks to help set expectations: - Small sites (under 500 pages): Typically 15 minutes to 2 hours - Medium sites (500–5,000 pages): Typically 2 to 5 hours - Large sites (5,000–50,000 pages): Typically 5 to 12 hours - Enterprise sites (50,000+ pages): Can exceed 12 hours in some cases Yes, an 8-hour audit is completely normal for a medium-to-large website. There is no need to cancel or restart the audit if it is still actively processing within these ranges. ⚙️ What Affects Audit Processing Time? Several factors can influence how long your site audit takes to complete: - Total number of pages: The more URLs your site has, the longer the crawl takes. - Server response speed: Slow-loading pages on your website mean the crawler has to wait longer for each response, which extends the overall audit time. - JavaScript-heavy pages: Sites that rely heavily on JavaScript rendering require additional processing time per page. - Crawl rate settings: If you have configured a conservative crawl rate to protect your server, the audit will run more slowly by design. - Platform load: During periods of high demand, audits across the platform may take slightly longer to process. ✅ How to Check Your Audit Status You can monitor the progress of your site audit at any time. Follow these steps: 1. In the left sidebar, click Site Explorer. 2. Select your site from the list to open the audit dashboard. 3. Look for the progress indicator at the top of the audit results page. It will show the percentage of pages crawled so far. 4. As long as the progress percentage is increasing, the audit is running normally — no action is needed. 🚫 When You Should Not Restart the Audit Avoid cancelling or restarting an audit that is still in progress, even if it seems to be taking longer than expected. Restarting will reset all progress and extend the total time needed to complete the audit. Only restart if: - The progress indicator has shown no change for more than 24 hours, or - The audit status displays an explicit error or failed message. 💡 Tips to Speed Up Future Audits If you want to reduce audit times going forward, consider these adjustments: - Exclude unnecessary URLs: Use the crawl settings to exclude parameters, duplicate pages, or non-essential subdirectories that do not need auditing. - Increase the crawl rate: If your server can handle more requests, raising the crawl rate in your audit settings will speed up the crawl. - Schedule audits during off-peak hours: Running audits overnight or during low-traffic periods can improve performance. 🛠️ Still Have Questions? If your audit has been stuck at the same progress percentage for more than 24 hours, or if you see an error message, our team is ready to help. If you need further assistance, open the chat widget in the bottom-right corner of the platform and type human teammate to be connected with a member of our team.

🛠️ Site Auditor Autopilot History and Export Reports

🔍 What Is Autopilot Deployment History? Site Auditor Autopilot automatically detects and fixes on-site SEO issues on your behalf. Every action it takes is logged so you always have a clear, auditable record of what was changed, when it happened, and what the outcome was. This history is available directly inside the platform without any additional setup. 📍 How to Find Your Autopilot Deployment History To view the history of automated changes Autopilot has made for a site, navigate to the Site Auditor section of the platform and open the Autopilot area for the project you want to review. From there, look for the history or log view, which displays a chronological record of every automated action Autopilot has completed for that site. 📋 What the Deployment History Shows Each entry in the deployment history log typically includes details such as: - When the change occurred — a timestamp for the automated action. - The type of SEO issue addressed — the category of problem that was fixed (for example, a missing meta description or a duplicate title tag). - The affected page — the URL where the fix was applied. - What action was taken — a description of what Autopilot changed or added. - The outcome — whether the fix was applied successfully or requires attention. You may be able to filter or sort the history to focus on specific time periods, issue types, or statuses. 📤 Exporting Your Autopilot Reports Search Atlas includes export functionality within the Site Auditor Autopilot area so you can share records of automated changes with clients, stakeholders, or your internal team. To generate an export, open the Autopilot history view for your project and look for an export or download option. The export will contain the logged details for the automated actions Autopilot has performed. If you cannot locate the export option or are unsure which report format is available for your plan, our support team can confirm the exact steps and available formats for your account. ❓ Troubleshooting and Next Steps If you cannot find the Autopilot history view, the export option is not visible, or the log does not appear to reflect recent changes, please have the following ready when you reach out: - The name of the project or site you are reviewing. - A description of what you expected to see versus what you are seeing. - Any error messages or unexpected behavior you have encountered. If you need further assistance, open the chat widget in the bottom-right corner of the platform and type human teammate to be connected with a member of our team.