Troubleshooting: Audits, Crawl & Indexing

🔍 Fix OTTO Crawler Failures Caused by Homepage Redirects

🛠️ Audit Logs, OTTO Sidebar, and Billing Updates

📋 What This Article Covers This guide answers three common questions in one place: how to review completed task history for your team members, what to do when OTTO SEO does not appear in your left sidebar, and how to update your billing email and phone number or change to the $399 plan. 📜 Reviewing Completed Tasks and Audit Logs Here is what is available today and what is still in development: - OTTO SEO audits: You can review audit history for your sites by going to Left sidebar → OTTO SEO → All Audits. This shows audits OTTO has run across your connected sites, so you can verify what work has been completed recently. - Billing activity: Your account activity log currently records billing-related actions, such as plan changes and payment updates. You can view this history in the Billing section of your account settings. - Per-member task logs: A full audit log that attributes every action to the specific team member who performed it is actively being developed. Until it ships, we are not able to generate a report of every task each member completed over the last two months. What you can do in the meantime: 1. Open Left sidebar → OTTO SEO → All Audits to confirm which sites have been audited and when. 2. Open Left sidebar → OTTO SEO → Overview for any site to see its current optimization status and recent OTTO activity. 3. If you need billing-specific history, our team can pull billing activity records for your account. If you need further assistance, open the chat widget in the bottom-right corner of the platform and type human teammate to be connected with a member of our team. 🔎 OTTO SEO Missing From Your Sidebar OTTO SEO lives in the left sidebar under AI SEO → OTTO SEO, and you can also reach it directly at the URL /seo-automation-v3. If you do not see it, work through these checks: 1. Expand the sidebar. If the left menu is collapsed, click the expand icon so all menu sections and labels are visible. 2. Scroll the full menu. OTTO SEO sits inside the AI SEO section. Its sub-pages include All Sites, Overview, Crawl Monitoring, Knowledge Base, Installation Guide, and All Audits. 3. Confirm your plan includes OTTO SEO. OTTO SEO is only visible on subscriptions that include it. Check your plan details under your account Billing settings. 4. Check your user role. If you are a member of a team or agency workspace rather than an owner or admin, your access may be limited. Ask your account owner to confirm your permissions. 5. Hard refresh your browser. Press Ctrl+Shift+R on Windows or Cmd+Shift+R on Mac, then try the direct URL /seo-automation-v3. If OTTO SEO still does not appear after these steps, your account may need an entitlement refresh on our side. If you need further assistance, open the chat widget in the bottom-right corner of the platform and type human teammate to be connected with a member of our team. 💳 Updating Your Billing Email and Phone Number Your billing contact details control where invoices and payment notifications are sent. To update them: 1. Click your profile or account icon and open Billing (or Account Settings → Billing). 2. Select the billing contact or payment details section. 3. Update the email address and phone number fields with your new information. 4. Save your changes. You may be asked to confirm the update for security reasons. Once saved, all future invoices and billing alerts will go to the new contact details. If you do not see editable contact fields in your Billing settings, our team can update them for you. If you need further assistance, open the chat widget in the bottom-right corner of the platform and type human teammate to be connected with a member of our team. ⬇️ Downgrading to the $399 Plan To change your subscription to the $399 plan: 1. Open Billing from your account settings and go to the Subscription or Plan section. 2. Select Change Plan (or Compare Plans) and choose the $399 plan. 3. Review the plan details, confirm the change, and save. Keep the following in mind: - Timing: Depending on where you are in your billing cycle, the downgrade may take effect immediately with a prorated credit, or at the start of your next cycle. - Feature access: Any features or usage limits above the $399 plan, such as extra projects or OTTO capacity, may be paused or reduced once the change takes effect. - Need help? If the $399 option is not visible, or you want to confirm how the change affects your current sites, our team can process the downgrade for you. If you need further assistance, open the chat widget in the bottom-right corner of the platform and type human teammate to be connected with a member of our team. 💬 Still Need Help? Whether it is audit history, a missing OTTO menu, or a billing change, our team is ready to assist. If you need further assistance, open the chat widget in the bottom-right corner of the platform and type human teammate to be connected with a member of our team.

🔄 Update OTTO Sitemap and Robots.txt After Deleting URLs

🗺️ Overview When you delete URLs from OTTO, three things need to happen in the correct order: your sitemap and robots.txt must reflect the changes, Google Search Console must be notified, and OTTO must complete a new analysis cycle to recognise the updated state of your site. This article walks you through each step. ✅ Step 1: Confirm Your Sitemap and Robots.txt Are Updated Before touching Google Search Console, verify that the files on your server already reflect the deletions. - Sitemap: Open your sitemap URL (typically yourdomain.com/sitemap.xml) and confirm the deleted URLs no longer appear. - Robots.txt: Open yourdomain.com/robots.txt and check that you have not accidentally blocked pages you still want indexed. If OTTO manages your robots.txt, any rules you applied inside OTTO should already be saved. If you edited the file manually, make sure the file is live on your server before continuing. 🤖 Step 2: Configure the SearchAtlasBot User-Agent in Robots.txt OTTO uses its own crawler called SearchAtlasBot to audit your site. For OTTO to crawl and analyse your pages accurately, this user-agent must not be blocked in your robots.txt file. Check your robots.txt for any Disallow rules that apply to SearchAtlasBot or to all bots via a wildcard (User-agent: *). A correctly configured entry looks like this: - User-agent: SearchAtlasBot - Disallow: (leave blank to allow full access) If SearchAtlasBot is blocked, OTTO will not be able to verify that your deleted pages are gone, and its analysis will not update correctly. Save and publish the robots.txt file once you have confirmed this entry. 📡 Step 3: Submit Your Updated Sitemap to Google Search Console 1. Log in to Google Search Console and select the property that matches your domain. 2. In the left sidebar, click Sitemaps under the Indexing section. 3. If your sitemap is already listed, click on it and then click Resubmit. If it is not listed, enter your sitemap URL in the Add a new sitemap field and click Submit. 4. Google will queue your sitemap for recrawling. The status will show as Success once it has been read. 🔍 Step 4: Request a URL Inspection and Recrawl in Google Search Console Submitting the sitemap tells Google your site has changed, but for priority pages you want deindexed quickly, use the URL Inspection tool to speed up the process. 1. In Google Search Console, paste a deleted URL into the URL Inspection search bar at the top and press Enter. 2. Once the inspection loads, click Request Indexing. 3. Google will add the URL to its crawl queue. This typically signals Google to check whether the page still exists and update its index accordingly. Repeat this for any high-priority deleted URLs. Note that Google processes these requests within hours to a few days, depending on your site's crawl budget and authority. ⏱️ Step 5: Understand OTTO Recrawl Timing and the Next Analysis Cycle After you update your sitemap, robots.txt, and Google Search Console, OTTO needs to run a new analysis cycle before it reflects the changes in your dashboard. - OTTO triggers a new crawl and analysis based on a scheduled cycle. Most sites are re-analysed within 24 to 48 hours after changes are detected or manually triggered. - If you want to prompt OTTO to re-analyse sooner, open your OTTO project, navigate to OTTO SEO → Site Audit → Overview (Website Overview), and use the Recrawl Site button. - Until the new cycle completes, previously deleted URLs may still appear in OTTO's issue list. This is expected behaviour and will resolve once the crawl finishes. 🔎 Step 6: Verify Page Status Using Page Explorer Once OTTO has completed its next analysis cycle, use the Page Explorer tool to confirm that your deleted URLs are no longer tracked as active pages. 1. Inside your OTTO project, open the Page Explorer tab. 2. Search for one of the deleted URLs using the search or filter bar. 3. Check the page's status. A deleted page should show a 404 or removed status, or it may no longer appear in the list at all. 4. If a deleted URL still shows as 200 OK, the page may still be live on your server. Double-check that the deletion was fully published and that any caching layers (CDN, server cache) have been cleared. Page Explorer gives you a ground-truth view of what OTTO's crawler sees, making it the most reliable way to confirm that your deletions have been processed correctly end to end. 📋 Quick Reference: Full Workflow Checklist - Confirm deleted URLs are removed from your live sitemap. - Verify SearchAtlasBot is not blocked in robots.txt. - Resubmit or submit your sitemap in Google Search Console. - Use URL Inspection to request recrawl for priority deleted URLs. - Wait 24–48 hours for OTTO's next analysis cycle to complete. - Check Page Explorer to confirm deleted pages show a 404 or removed status. 💬 Need More Help? If you need further assistance, open the chat widget in the bottom-right corner of the platform and type human teammate to be connected with a member of our team.

🔄 OTTO Recrawl vs Weekly Analysis Explained

🤔 What Is the Difference Between a Recrawl and a Weekly Analysis? OTTO runs two distinct processes to evaluate your site's SEO health: a manual recrawl that you trigger on demand, and an automated weekly analysis that runs in the background on a fixed schedule. Understanding what each one does helps you use OTTO more effectively after making changes to your site. 📅 How the Weekly Analysis Works OTTO automatically re-analyses your site once every seven days. This cycle is designed to keep your SEO recommendations fresh without requiring any action from you. During a weekly analysis, OTTO: - Re-evaluates all previously identified SEO issues across your site - Detects new issues that may have appeared since the last cycle - Updates recommendation statuses based on the latest state of your pages - Refreshes performance data and opportunity scores Because the weekly analysis runs on a set schedule, it is best suited for routine monitoring rather than verifying a specific fix you have just deployed. 🔍 What a Manual Recrawl Does A manual recrawl lets you immediately re-examine your site after you have made code or content changes. Instead of waiting up to seven days for the next scheduled cycle, you can trigger OTTO to crawl your site right away and confirm whether your fixes have resolved the flagged recommendations. During a manual recrawl, OTTO: - Re-crawls the pages associated with active recommendations - Checks whether previously flagged issues are now resolved - Updates the status of affected recommendations in real time - Reflects any structural or on-page changes you have implemented A manual recrawl is scoped to your existing recommendations — it validates fixes rather than performing a full discovery sweep of brand-new issues. For a comprehensive discovery of new issues, the weekly analysis cycle handles that automatically. 🛠️ How to Trigger a Manual Recrawl in OTTO 1. In the left sidebar, click OTTO SEO, then navigate to Site Audit → Overview (Website Overview). 2. Locate the site session you want to recrawl. 3. Click the Recrawl Site button associated with that session. The button is typically displayed near the top of the session overview or within the site's action menu. 4. OTTO will begin crawling immediately. Depending on your site's size, this may take a few minutes to complete. 5. Once the recrawl finishes, return to your recommendations list to review updated statuses and confirm that your fixes have been picked up. ✅ When Should You Use Each Option? - Use a manual recrawl when you have just pushed code changes, updated meta tags, fixed broken links, or resolved any specific recommendation and want immediate confirmation. - Rely on the weekly analysis for ongoing site health monitoring, discovering new SEO opportunities, and tracking overall progress over time without manual intervention. 💡 Tips for Getting Accurate Recrawl Results - Make sure your code changes are fully deployed and live before triggering a recrawl. OTTO crawls the live version of your site, so any changes still in staging will not be reflected. - Clear your CDN or server cache before recrawling to ensure OTTO reads the most up-to-date version of your pages. - Allow the recrawl to complete fully before reviewing results — checking mid-crawl may show incomplete data. 💬 Need More Help? If you need further assistance, open the chat widget in the bottom-right corner of the platform and type human teammate to be connected with a member of our team. Additional Notes Immediate recrawl does NOT detect: deleted/redirected pages, codebase-level modifications, new pages added, site architecture/navigation changes, server-side rendering/framework-level changes. Results typically reflected within a few minutes to a couple of hours depending on site size.

⚙️ Fix OTTO GSC Instant Indexing

🔍 What the error means The Plan insufficient message means your current Search Atlas subscription does not include OTTO GSC Instant Indexing. The feature cannot be enabled until your account has an eligible plan or add-on. ✅ Check your plan 1. Open the account menu from your avatar in the top-right corner. 2. Select Billing, then open the Plans & Top-ups tab. 3. Review your current subscription and available plans or top-ups. 4. Upgrade or add the required service if GSC Instant Indexing is not included. 🔗 Connect Google Search Console After confirming that your plan supports the feature, make sure the correct Google Search Console property is connected to the project. OTTO cannot activate GSC Instant Indexing without an active GSC connection and the required permissions. 1. Go to Site Metrics → Site Explorer (Manage Sites → Connect GSC Account). 2. Reconnect GSC if it is disconnected, then select the correct property. 3. Return to OTTO and enable GSC Instant Indexing again. 🔄 If the error continues Refresh the page after reconnecting GSC and try again. If the feature still shows Plan insufficient after your plan has been updated, sign out and back in to refresh account access and billing status. If you need further assistance, open the chat widget in the bottom-right corner of the platform and type human teammate to be connected with a member of our team.

🔍 Fix False-Positive Audit Flags in OTTO

🧐 Why Am I Seeing These Flags? When OTTO's Site Audit reports Multiple Meta Descriptions or Too Many External Links on a page you know is clean, you are seeing a false positive. These flags are not generated in real time. They reflect the data captured during the last completed analysis crawl. If your page had a temporary issue at the time of that crawl — such as a caching layer serving a duplicate meta description, a plugin injecting extra links, or a deployment that has since been rolled back — the flag will remain visible until the next crawl refreshes the data. In short: OTTO reports what it found during its last crawl, not the current live state of your page. 📋 Common Reasons for Each Flag - Multiple Meta Descriptions: A caching plugin or CDN served a cached page with a duplicate meta tag at crawl time. A theme or SEO plugin was briefly outputting two meta description tags. A staging-to-live deployment temporarily introduced conflicting tags. - Too Many External Links: A widget, footer script, or ad network was active at crawl time and injected additional outbound links. A temporary plugin or A/B test added links that have since been removed. A page template included partner links that were later cleaned up. Because OTTO evaluates the HTML it receives during the crawl, any transient condition present at that moment will be recorded and flagged — even if you fix the issue immediately afterward. ⏱️ When Will the Flags Clear? Flags clear automatically on the next scheduled weekly analysis. OTTO runs a full site analysis on a weekly cycle. Once that analysis completes and no longer detects the issue on your page, the flag will be removed from your audit results. Important: Triggering a manual recrawl or reprocessing a page inside the platform does not refresh Site Audit flags. A manual recrawl updates other data points but does not re-run the full audit analysis that controls these issue flags. Only the weekly analysis cycle does that. ✅ What You Should Do Right Now 1. Open your page in a browser and use View Page Source (right-click → View Page Source) to confirm only one meta description tag is present in the raw HTML. 2. Check your page in an incognito window and via a tool such as Google's Rich Results Test to rule out caching serving a different version to crawlers. 3. Review any active plugins, widgets, or scripts that could inject external links — disable them temporarily if needed and verify the source again. 4. If the page source is clean, no further action is required. The flag will clear automatically after the next weekly analysis runs. 5. Make a note of the date so you can confirm the flag is gone after the next weekly cycle completes. 🔄 How to Check Your Audit Results You can review current Site Audit flags by navigating to Left sidebar → OTTO SEO → Site Audit and opening your OTTO dashboard. Flags are listed under the relevant issue categories. The date of the last completed analysis is displayed at the top of the audit results so you can confirm how recent the data is. ❓ What If the Flag Persists After the Next Weekly Analysis? If the flag is still present after a full weekly analysis has run and your page source is confirmed clean, the issue may be genuine and related to how a crawler or bot sees your page differently from a standard browser. In that case: - Test your URL using a crawler-emulation tool to see what a bot receives. - Check whether your server or CDN is serving different HTML to user agents identified as crawlers. - Review any server-side rendering or JavaScript frameworks that may inject tags or links after the initial HTML is delivered. If you need further assistance, open the chat widget in the bottom-right corner of the platform and type human teammate to be connected with a member of our team.

🤖 Managing Noindex and Robots.txt Without Backend Access

Understanding the Constraint When a client implements OTTO through Google Tag Manager (GTM) rather than the WordPress plugin, you typically do not have direct access to the server's robots.txt file or the site's CMS backend. This means standard approaches — editing robots.txt directly or adding meta tags through a CMS admin panel — may not be available to you. This is a known configuration gap. Because the specific steps for resolving this depend on your site setup and what access your client can grant, the most reliable path is to work directly with our support team, who can guide you through the appropriate approach for your situation. What to Have Ready Before Escalating To get the fastest resolution, please have the following information available when you contact us: - Your project name and the domain where OTTO is deployed - Confirmation of how OTTO is implemented (e.g., via GTM, WordPress plugin, or another method) - A description of what you are trying to achieve — for example, applying noindex to specific pages, blocking certain paths in robots.txt, or removing URLs from Google's index - The specific pages or URL patterns you need to control - What backend or CMS access you currently have (e.g., GTM container access, Google Search Console access, FTP/hosting access) - Any error messages or unexpected crawl/index behavior you have observed, including timestamps if available General Guidance Depending on your level of access, common approaches for this type of issue include collaborating with your client to apply changes at the server or CMS level, using any GTM or Search Console access you do have, or requesting that your client implement specific directives on your behalf. Our support team can help you identify which option fits your situation and walk you through the correct steps. If you need further assistance, open the chat widget in the bottom-right corner of the platform and type human teammate to be connected with a member of our team.

🔧 Fix Low OTTO Crawl Volume and Schedule

Overview If OTTO is indexing significantly fewer pages than your site contains—for example, 16,000 of 65,000 pages—this is a crawl volume issue, not a plugin installation problem. This article explains the most common root causes of low crawl coverage and how to diagnose them. Why Is OTTO Only Reaching a Subset of Pages? There are five primary reasons OTTO may index fewer pages than expected. Work through each one before assuming a technical fault. 1. Crawl budget limits: Large sites have a finite crawl budget—the number of URLs a crawler will fetch in a given period. If your site has 65,000 pages, OTTO prioritises the most discoverable and link-connected pages first. Pages buried deep in your architecture or receiving few internal links are reached in later crawl cycles. To expand coverage, improve internal linking to surface orphaned or deep pages. 2. Noindex tags on large page groups: Template-level noindex meta tags or HTTP headers can silently exclude entire categories, tags, paginated archives, or product filter URLs. Audit your page templates—especially e-commerce faceted navigation, paginated series, and auto-generated tag pages—to confirm intended pages are not marked noindex. 3. Robots.txt restrictions: A Disallow rule in your robots.txt file can block OTTO from crawling entire URL path segments. Review your robots.txt (accessible at yourdomain.com/robots.txt) and confirm that no rules accidentally block large sections of the site such as /products/, /blog/, or /category/. 4. Sitemap gaps: If your XML sitemap does not include all expected pages—or if it contains broken URLs, incorrect priorities, or is not submitted—OTTO has no reliable map to follow. Verify your sitemap is complete, up to date, and submitted within the platform. 5. Site structure and internal linking depth: Pages more than three to four clicks from the homepage are harder for any crawler to reach within a standard crawl budget window. Flatten your site architecture where possible and ensure key pages are reachable through prominent internal links. What to Have Ready When Escalating If you have reviewed all five areas above and crawl coverage remains unexpectedly low, please gather the following before contacting support: - Your project name and the domain being crawled - The total number of pages on your site and the number OTTO has indexed - A copy of your robots.txt file contents - Your XML sitemap URL - Any specific URL patterns or sections of the site that appear to be missing from the crawl If you need further assistance, open the chat widget in the bottom-right corner of the platform and type human teammate to be connected with a member of our team.

🤖 OTTO Crawler Robot Challenges and 202 Status Codes

🔍 Overview If you have deployed title tags or meta descriptions through OTTO and they appear to revert without any action on your part, the most likely cause is that the affected pages are returning a 202 status code instead of a standard 200 OK response. A 202 status means the server has accepted the request but has not yet finished processing the page. OTTO cannot fully crawl or confirm fixes on pages in this state. ⚙️ What Is a 202 Status Code? HTTP status codes tell crawlers how a server responded to a request: - 200 OK — The page loaded successfully and is fully crawlable. - 202 Accepted — The server received the request but the page is still being processed. The content is not yet in its final state. When OTTO encounters a 202 response, the page is not considered crawlable or indexable. As a result, OTTO cannot verify that your deployed fixes are live, and the dashboard may show those pages as unresolved or appear to show the original, unfixed metadata. 🤖 What Is a Robot Challenge Screen? A robot challenge screen is a security layer — commonly served by tools like Cloudflare — that presents an automated browser check before allowing access to a page. When OTTO's crawler encounters this challenge, it cannot read the page content and receives a non-200 response (such as a 202) instead of the finished HTML. This prevents OTTO from confirming or reading your deployed SEO changes. 🛠️ How to Diagnose the Problem 1. Identify which pages appear to have reverted titles or meta descriptions after deployment. 2. For each affected page, manually check the HTTP status code. You can do this by opening your browser's developer tools, going to the Network tab, and reloading the page — look for the status code shown next to the page URL. Alternatively, use a free online HTTP status checker by pasting in the page URL. 3. If a page returns 202, the page is still processing or is behind a bot-challenge screen. OTTO will not be able to confirm fixes on that page until it returns a 200 OK. 4. If a page returns 200 OK, OTTO should be correctly reading and displaying your deployed changes. If fixes still appear missing on 200-status pages, proceed to request a recrawl (see below). ✅ What You Can Do - Wait for processing to complete. If your pages are genuinely still loading or being built server-side, wait and check the status code again after some time. Once the page returns 200 OK, OTTO will be able to crawl and confirm your fixes. - Check your site's bot-protection settings. If your hosting provider or CDN (such as Cloudflare) has bot-fight or challenge modes enabled, these can block legitimate crawlers including OTTO. Review your security settings and, if safe to do so, configure exceptions to allow automated crawlers access to your pages. - Trigger a recrawl in OTTO. Once affected pages are returning 200 OK responses, initiate a recrawl from within the OTTO section of the platform so that OTTO can re-read and confirm your deployed fixes. - Do not redeploy unnecessarily. Redeploying fixes to pages still returning 202 responses will not resolve the underlying issue and may cause confusion. Wait until the pages are fully accessible before redeploying. 📊 Pages With 200 OK Status For any pages that are already returning a 200 OK status code, OTTO should be correctly displaying your original deployed titles and meta descriptions. If you notice discrepancies on those pages specifically, this may be a separate caching or deployment issue rather than a crawl-access problem. 💡 Additional Context This issue is most common on sites hosted on platforms with aggressive bot-protection enabled, or on pages that use server-side rendering with delayed processing. The root cause is always that OTTO cannot access the final, rendered version of the page — not that your SEO changes have been lost. Your deployed changes remain intact; OTTO simply cannot confirm them until the page is crawlable. 🆘 Still Need Help? If you have verified that your pages are returning 200 OK responses and your fixes are still not being reflected, or if you are unsure how to adjust your bot-protection settings, If you need further assistance, open the chat widget in the bottom-right corner of the platform and type human teammate to be connected with a member of our team.

🛠️ Fix Website Studio Sitemap and Indexing Errors

🔍 Overview If your Website Studio website is experiencing sitemap errors, missing pages, noindex flags, incorrect title tags on Google, or unexpected ranking drops, several known platform issues may be contributing. This article explains each problem and the steps you can take to resolve them. 🗺️ Sitemap Errors and Missing Homepage Your sitemap may be incomplete or contain errors for a few reasons: - Draft or unpublished pages leaking into the sitemap: Previously, unpublished draft versions of your site could be crawled by search engines. This has been resolved, but if Google cached an earlier version of your sitemap, it may still show stale data. - Preview and editor URLs in the sitemap: Internal editor, preview, and generated URLs were sometimes included in the live sitemap, polluting it with non-canonical addresses. This issue has been fixed. However, you should manually verify your sitemap is clean. - Homepage missing from sitemap: This can happen if the homepage was not set as published in Website Studio, or if a redirect chain is preventing Google from resolving it correctly. To check and fix your sitemap: 1. Log in to Search Atlas and open Website Studio. 2. Confirm your homepage is published and set as the primary page. 3. Navigate to your live sitemap URL (typically yourdomain.com/sitemap.xml) and review every URL listed. 4. Remove or unpublish any draft, preview, or duplicate pages that appear in the list. 5. Request a sitemap re-fetch via Google Search Console under Sitemaps to prompt Google to re-crawl the updated version. 🚫 Noindex Flags on Pages Noindex tags tell search engines not to index a page. If your pages carry unintended noindex flags, they will not appear in search results regardless of how well-optimised they are. Common causes in Website Studio include: - Preview or staging versions of the site being served with noindex headers, which can persist if the domain was not correctly transitioned to live. - Individual page settings accidentally toggled to noindex during editing. To check for and remove noindex flags: 1. Open Website Studio and go to each key page's SEO Settings. 2. Confirm the Allow search engines to index this page option is enabled. 3. Use the Site Audit tool in Search Atlas to scan for pages returning noindex signals. 4. For any page flagged as noindex, update the setting and republish the page. 5. Verify the fix by entering your page URL into Google Search Console's URL Inspection tool. 🏷️ Incorrect Title Tags Displaying on Google If Google is showing a different title than the one you set in Website Studio, this is usually caused by one of the following: - Google rewrites titles when it determines the set title does not accurately represent the page content. This is a Google-side behaviour and not always within your control. - Duplicate or competing pages: If preview URLs or draft pages were previously indexed, Google may be pulling the title from a cached version of one of those pages instead of your live page. - Title tag not saved correctly: A publish step may have been missed after updating the title in Website Studio. To correct title tag display issues: 1. In Website Studio, open the page editor and navigate to SEO Settings. 2. Confirm the title tag is accurate, relevant, and between 50–60 characters. 3. Save and republish the page. 4. In Google Search Console, use URL Inspection and select Request Indexing to prompt Google to recrawl the updated page. 5. If old preview or draft URLs were previously indexed, request their removal using the Removals tool in Google Search Console. 📉 Poor Rankings After Persistent Issues Months of sitemap errors, noindex flags, and competing preview URLs can negatively affect rankings even after the underlying issues are fixed. Recovery takes time because Google needs to recrawl and re-evaluate your site. To accelerate recovery: - Ensure all preview and draft domains are fully blocked from indexing. Preview domains serving sitemap or content files have now been addressed at the platform level, but confirm none remain accessible. - Submit your corrected sitemap in Google Search Console. - Use the Site Audit tool in Search Atlas to identify any remaining crawl errors, redirect issues, or duplicate content problems. - Build or refresh internal links pointing to your most important pages to reinforce their priority to search engines. - Monitor Google Search Console weekly to track improvements in crawl coverage and impressions. 📄 Blog Pages Displaying Raw XML If a blog article published through Website Studio displays raw XML in the browser instead of formatted content, this is a known rendering issue that has been resolved at the platform level. If you still encounter this on any article: 1. Open the article in Website Studio. 2. Unpublish the article, then republish it to trigger a fresh render. 3. Clear any caching on your domain if you use a CDN or caching plugin. 4. Check the live URL in a private browser window to confirm the issue is resolved. 💬 Need More Help? If you need further assistance, open the chat widget in the bottom-right corner of the platform and type human teammate to be connected with a member of our team.

⚙️ Core Web Vitals Not Updating in OTTO Dashboard

🔍 Overview Some customers have noticed that Core Web Vitals scores in the OTTO dashboard have not refreshed, even after rescanning their site or making performance improvements. This article explains why this happens and what you can do in the meantime. ⚠️ What Is Happening Core Web Vitals data inside the OTTO dashboard is sourced from the Chrome User Experience Report (CrUX), which is Google's real-user field data collected over a 28-day rolling window. Our engineering team has identified a bug where this CrUX data is not refreshing as expected, even when a new Site Audit crawl is triggered. As a result, the scores displayed in your OTTO dashboard may appear frozen or out of date. Our team is actively working on a fix, and a resolution is being prioritised. 📊 Why Rescanning Does Not Force a Refresh It is important to understand that triggering a new Site Audit crawl does not automatically pull fresh CrUX field data from Google. The two processes are separate: - Site Audit crawl — checks your site's technical structure, links, and on-page elements. - Core Web Vitals (CrUX field data) — pulls real-user experience data that Google aggregates over a rolling 28-day period. This data updates on Google's own schedule and is fetched separately by our platform. Because of the current bug, our platform is not successfully retrieving the latest CrUX dataset even when a refresh is attempted, so the dashboard continues to show older scores. 🛠️ What to Do Right Now While the fix is being deployed, you can still monitor your Core Web Vitals performance using the following alternatives: 1. Google PageSpeed Insights — Visit pagespeed.web.dev and enter your URL to get both lab data and field data directly from Google. This gives you the most up-to-date CrUX scores available. 2. Google Search Console — Check the Core Web Vitals report in your Search Console account. This report reflects real-user field data for your entire site. 3. Chrome DevTools or Lighthouse — Run a local Lighthouse audit for lab-based performance metrics that reflect your current site state regardless of CrUX availability. 💡 What to Expect After the Fix Once the engineering fix is fully deployed, the OTTO dashboard will resume pulling fresh CrUX field data on its normal schedule. At that point, Core Web Vitals scores will update to reflect the most recent 28-day rolling window of real-user data, and any improvements you have made will be reflected in the refreshed scores once enough real-user traffic has been recorded by Google. If you need further assistance, open the chat widget in the bottom-right corner of the platform and type human teammate to be connected with a member of our team.

🔍 OTTO Crawl Failures: Fixing Homepage Redirect Issues

🧭 Overview When OTTO's crawler cannot successfully reach your homepage, it fails to initialize a site scan — even if your OTTO pixel is correctly installed in your source code. This creates a confusing situation where your pixel appears in the page HTML but OTTO still reports it as undetected. The root cause is almost always a redirect configuration that blocks the crawler before it can verify the pixel. ⚠️ Why OTTO Requires a Directly Accessible Homepage URL OTTO's crawler starts every site scan by fetching your homepage URL. For the scan to succeed, that URL must return a valid response without passing through a problematic redirect chain. If the crawler follows a redirect that leads to an unexpected destination, it may fail to complete the crawl or detect your pixel. Common redirect patterns that can cause this failure include: - HTTP → HTTPS redirects where the URL saved in OTTO uses http:// but your server forces https:// - Non-www → www redirects (or vice versa) where the saved URL does not match the canonical version of your homepage - Custom path redirects where the root domain redirects to a subdirectory such as /home or /en/ - Redirect chains combining two or more of the above 🔗 How Redirect Chains Block Crawler Initialization Each redirect hop adds latency and risk. When a redirect chain leads to a URL outside the registered domain, or otherwise prevents the crawler from reaching the page, the scan may fail. Because the scan never completes, OTTO cannot read your page source — which is why your pixel appears in the source code when you check manually but remains undetected inside the platform. Think of it this way: you installed the pixel correctly, but OTTO never got close enough to the page to see it. 🩺 Step-by-Step Diagnosis 1. Find the URL registered in OTTO. Navigate to your OTTO project and locate the homepage URL that was entered during setup. Note the exact URL, including the protocol (http or https) and whether it includes or excludes www. 2. Check what your homepage actually resolves to. Open a new browser tab and paste your registered URL into the address bar. Watch where the browser ends up after all redirects complete. Copy the final URL from the address bar. 3. Compare the two URLs. If the registered URL and the final resolved URL are different in any way — protocol, subdomain, trailing slash, or path — you have likely found the source of the crawl failure. 4. Update the registered URL in OTTO to match the final resolved URL exactly, so the crawler can reach your homepage without following any redirects. 5. Re-trigger the site scan after updating the URL and confirm that OTTO successfully detects your pixel. If you need further assistance, open the chat widget in the bottom-right corner of the platform and type human teammate to be connected with a member of our team.

⚡ OTTO Recrawl Scheduling and Audit Issue Navigation

Overview OTTO performs automatic site audits on a recurring schedule so you always have fresh SEO data without manual intervention. This article explains what to expect from OTTO's recrawl process and how to locate specific audit issues — such as Page not in sitemap — inside your audit results. How Automatic Recrawl Scheduling Works OTTO is designed to run site audits automatically for your project. You generally do not need to trigger recrawls manually under normal circumstances. Here is what to expect from the automatic recrawl process: - OTTO initiates audit crawls on a recurring schedule managed by the platform. - When a crawl completes, OTTO processes the results and updates your audit data automatically. - Refreshed issue counts, affected page totals, and recommendations will appear once the crawl finishes — no action is required on your part. - If your audit data appears stale or not yet updated, allow some time after the scheduled crawl window before concluding data has not refreshed, as processing can take time depending on your site's size. How to Find Specific Audit Issues Issues such as Page not in sitemap are grouped and listed within your OTTO audit results. To locate a specific issue: 1. In the left sidebar, go to OTTO SEO → Site Audit. 2. Select your project to open its Site Audit results. 3. Browse or search the list of flagged issues to find the one you are looking for, such as Page not in sitemap. 4. Select the issue to view the full list of affected pages and any available guidance on addressing it. If an issue you expect to see does not appear, the crawl may still be processing. Wait for the audit to complete and then refresh your view. What to Do If You Need Help If your audit data is not updating, a specific issue is missing from your results, or you are unsure how to interpret your audit findings, please reach out to our support team with the following information ready: - Your project name or URL - A description of the issue you are seeing (e.g., missing audit results, unexpected issue counts) - Any relevant timestamps or screenshots If you need further assistance, open the chat widget in the bottom-right corner of the platform and type human teammate to be connected with a member of our team.

🛠️ Fix OTTO Connection Stuck in Processing or Crawling

🔍 What This Issue Looks Like After installing the OTTO script on your website and attempting to connect your site, the connection process does not complete. The status remains stuck — either in a processing or crawling state — even after waiting several hours or days. The site never fully activates in OTTO. ⚠️ Why This Happens In some cases, OTTO's background crawl process can stop silently without returning an error message. The platform appears to be working, but the crawl has actually stalled internally. This is a known issue that has been identified and addressed by the Search Atlas engineering team. However, if your site is currently stuck, the steps below can help you resolve it. ✅ How to Fix It 1. Remove the OTTO script from your website. Log in to your website's backend and delete the OTTO script that was installed during the connection setup. 2. Delete the existing OTTO project. In the Search Atlas platform, go to Left sidebar → OTTO SEO → All Sites (SEO Automation) and remove the project associated with your stuck site. 3. Wait a few minutes before proceeding to allow the system to fully clear the previous connection attempt. 4. Create a fresh OTTO project. Still in Left sidebar → OTTO SEO → All Sites (SEO Automation), create a new project for your website as if connecting it for the first time. 5. Reinstall the OTTO script. Follow the setup instructions to install the OTTO script on your site again, using the installation steps provided within the OTTO SEO section of the platform. 6. Trigger a new connection. Once the script is reinstalled, attempt to connect your site again and monitor the status over the next 15–30 minutes. 📊 After Reconnecting If the connection completes successfully, your site will move past the processing stage and OTTO will begin its full setup. You can monitor the progress from the OTTO SEO section of the platform. 🚨 If the Issue Persists The fresh-connection approach resolves this issue in most cases. However, if your site remains stuck after following all the steps above, this may indicate a deeper infrastructure issue that requires engineering intervention — similar to cases where the crawl dies silently and cannot self-recover. If you need further assistance, open the chat widget in the bottom-right corner of the platform and type human teammate to be connected with a member of our team. When you reach out, please have the following information ready to help the team investigate quickly: - Your website URL - The date and time you first attempted to connect - Confirmation that you have already removed and reinstalled the script - A description of the curr

🛠️ Fix OTTO Integration Errors and Site Metrics Crawling

🔍 Overview If OTTO is showing an integration error on your website, or if Site Explorer is not crawling your domain correctly, this guide walks you through the most common causes and how to fix them quickly. Most issues can be resolved in just a few steps without needing to contact support. ⚠️ Common Causes of OTTO Integration Errors Before troubleshooting, it helps to understand why the error may be occurring. The most frequent causes include: - Incorrect website URL: The domain added to OTTO does not exactly match the live site URL (e.g., missing www or using http instead of https). - OTTO code not installed: The OTTO snippet has not been placed correctly on your website. - Crawl not yet completed: Site Explorer needs time to finish its initial crawl before data appears. - Domain verification failure: Search Atlas cannot confirm ownership or access to the domain. - Site access restrictions: Password protection, firewall rules, or bot-blocking settings may be preventing the crawler from reaching your site. 🛠️ Step 1 — Check Your OTTO Integration Setup 1. In the left sidebar, click OTTO SEO, then open SEO Automation (URL: /seo-automation-v3). 2. Locate the website showing the integration error and click into its OTTO dashboard. 3. Verify that the domain listed matches your live website URL exactly, including the correct protocol (https://) and subdomain (www or non-www). 4. If the URL is incorrect, remove the site and re-add it using the exact URL format your website uses. 💡 Step 2 — Confirm the OTTO Snippet Is Installed 1. Inside the OTTO dashboard for your site, locate the integration code snippet provided by Search Atlas. 2. Confirm that this snippet has been added to every page of your website, ideally in the section of your site template. 3. If you are using a CMS such as WordPress, verify the snippet was not removed during a recent theme update or plugin change. 4. After confirming the snippet is in place, click Verify Integration (if available) or allow up to 24 hours for the system to re-detect it automatically. 📊 Step 3 — Check Site Explorer Crawl Status 1. In the left sidebar, click Site Metrics → Site Explorer (URL: /site-explorer/list). 2. Find your domain in the list and check its crawl status. 3. If the crawl is shown as In Progress, wait for it to complete — initial crawls can take up to 24 hours depending on site size. 4. If the crawl shows an error or has been stuck for more than 24 hours, remove the domain from Site Explorer and re-add it to trigger a fresh crawl. 🔒 Step 4 — Remove Site Access Restrictions If your website has any of the following in place, the OTTO crawler and Site Explorer crawler may be blocked: - Password-protected pages or staging environments - Cloudflare or similar CDN bot-protection rules blocking crawlers - robots.txt rules that disallow all bots - Firewall or security plugin settings that block unrecognised user agents Temporarily disable these restrictions, trigger a new crawl, then re-enable them once the integration is confirmed. If you need to whitelist the Search Atlas crawler specifically, contact our team via the chat widget for the current crawler IP addresses and user agent details. 🔄 Step 5 — Re-Trigger the Integration Check 1. After completing the steps above, return to OTTO SEO in the left sidebar and open SEO Automation. 2. Open the affected site's OTTO dashboard. 3. Look for a Retry or Re-check Integration option and click it. 4. Allow up to 30 minutes for the system to confirm the integration. Refresh the page after this period to see the updated status. ✅ When Everything Is Working Once the integration is active and the crawl has completed, you will see: - A green connected status on the OTTO dashboard under OTTO SEO. - Populated site data and recommendations inside OTTO. - Crawl results visible under Site Explorer, with sub-sections such as Keyword Magic, Rank Tracker, and Keyword Gap showing data for your domain. 💬 Still Seeing an Error? If you have followed all the steps above and the integration error or crawling issue persists, our support team is ready to investigate further. If you need further assistance, open the chat widget in the bottom-right corner of the platform and type human teammate to be connected with a member of our team.

🕷️ Fix OTTO Crawler Installation and Whitelisting Issues

🔍 Overview After installing OTTO SEO, some customers notice that the crawler only indexes a single page — typically the homepage — instead of crawling the entire website. This usually happens because a security plugin, firewall, or server-level rule is blocking the Search Atlas crawler bot after its first request. This article walks you through how to identify the cause and whitelist the crawler correctly so OTTO can index your full site. ⚙️ How the OTTO Crawler Works When OTTO is activated for your website, the Search Atlas crawler visits your pages to gather SEO data and apply optimizations. If anything on your server interprets repeated bot requests as suspicious activity, it may block the crawler after the first visit — leaving OTTO with only partial data to work with. Common causes of this issue include: - Security plugins (e.g., Wordfence, Sucuri, iThemes Security) blocking unrecognized bots - Cloudflare or CDN firewall rules that rate-limit or challenge bot traffic - Hosting-level bot protection that restricts crawlers automatically - robots.txt rules that unintentionally disallow the Search Atlas crawler 🛠️ Step 1 — Whitelist the Search Atlas Crawler User-Agent The Search Atlas crawler identifies itself with the following user-agent string: SearchAtlas/1.0 You need to add this user-agent to the allowlist in any tool that filters bot traffic on your site. Follow the steps below based on your setup. 🔒 Step 2 — Whitelist in Your Security Plugin If you are using a WordPress security plugin, here is how to allow the crawler: 1. Log in to your WordPress dashboard. 2. Navigate to your security plugin settings (e.g., Wordfence → Firewall → Allowlisted IPs or Bots). 3. Locate the option to whitelist a bot or user-agent. 4. Add SearchAtlas/1.0 as a trusted user-agent. 5. Save your changes. If your plugin does not support user-agent whitelisting, check whether it allows you to whitelist by IP address. Contact our support team via the chat widget (see below) to request the current list of Search Atlas crawler IP addresses. ☁️ Step 3 — Whitelist in Cloudflare or Your CDN If your site uses Cloudflare or another CDN with bot management, follow these steps: 1. Log in to your Cloudflare dashboard and select your domain. 2. Go to Security → WAF → Custom Rules (or Firewall Rules, depending on your plan). 3. Create a new rule that allows traffic where the user-agent contains SearchAtlas. 4. Set the action to Allow and save the rule. 5. If you use Cloudflare Bot Management or Super Bot Fight Mode, ensure verified bots or the Search Atlas user-agent are not being challenged or blocked. 📄 Step 4 — Check Your robots.txt File Your robots.txt file may be blocking the Search Atlas crawler. To check: 1. Open a browser and go to yourdomain.com/robots.txt. 2. Look for any Disallow rules under a wildcard agent (User-agent: *) that might block large portions of your site. 3. If needed, add a specific entry to explicitly allow the Search Atlas crawler: User-agent: SearchAtlas/1.0 Allow: / Add these lines above any wildcard disallow rules so they take priority. Save and re-upload the file to your server root if you edit it manually. 🚀 Step 5 — Re-Trigger the OTTO Crawl Once you have completed whitelisting, you need to prompt OTTO to re-crawl your site: 1. In Search Atlas, go to OTTO SEO in the left sidebar, then open SEO Automation. 2. Select the website where OTTO is installed. 3. Locate the crawl or re-crawl option and initiate a new crawl. 4. Wait for the crawl to complete — this may take a few minutes depending on your site size. 5. Review the crawled pages count to confirm all pages are now being discovered. ✅ Confirm the Issue Is Resolved After the re-crawl, check the following to confirm everything is working: - The total pages crawled matches the expected number of pages on your site. - OTTO is surfacing SEO recommendations across multiple pages, not just the homepage. - No new error messages appear in the OTTO dashboard related to crawl access. If the crawled page count is still lower than expected, double-check that all layers of protection on your site — including hosting-level firewalls and CDN rules — have been updated. 💬 Still Need Help? If you need further assistance, open the chat widget in the bottom-right corner of the platform and type human teammate to be connected with a member of our team.

🕷️ OTTO Only Crawling One Page After Installation

🔍 Overview After installing OTTO on your website, the crawler should systematically visit and index all pages within your connected domain. If OTTO reports only 1 page crawled after installation, your server or security layer is almost certainly blocking the Search Atlas crawler before it can follow internal links. This article walks you through diagnosing the block and whitelisting the crawler on the most common platforms. ⚠️ Symptoms That Confirm Crawler Blocking - OTTO shows 1 page crawled and the count does not increase over time. - The crawl appears to complete unusually fast (within seconds). - Internal links on your site are not discovered or are shown as uncrawled. - Your server logs show repeated 403, 429, or 503 responses from the Search Atlas crawler user-agent or IP range. - You have recently added or updated a firewall, WAF, CDN, or bot-protection rule. If you see one or more of these signs, proceed to the diagnosis steps below. 🛠️ Step 1 — Identify What Is Blocking the Crawler Work through this quick diagnosis tree to pinpoint the source of the block before attempting a fix. 1. Check your robots.txt file. Visit yourdomain.com/robots.txt and look for any Disallow rules that could apply to all bots or to the Search Atlas user-agent specifically. If a broad Disallow: / rule exists, the crawler cannot access any page beyond the entry point. 2. Check for a WAF or bot-protection service. If your site uses Cloudflare, Sucuri, Wordfence, AWS WAF, Imperva, or a similar tool, its bot-blocking rules may be treating the Search Atlas crawler as a threat and returning a block page after the first request. 3. Check your hosting control panel. Some managed hosts (WP Engine, Kinsta, SiteGround, etc.) include built-in bot filtering that can block crawlers not on their approved list. 4. Check for rate-limiting rules. A rate limiter may allow the first request (the home page) and then drop all subsequent requests from the same IP, causing OTTO to stall at exactly 1 page. ✅ Step 2 — Whitelist the Search Atlas Crawler Once you know where the block is coming from, use the relevant instructions below to allow the crawler through. Search Atlas Crawler IP Addresses Add the following IP addresses to your allowlist in whichever platform is blocking the crawler. Always refer to your OTTO dashboard or contact our team for the most current IP list, as addresses may be updated periodically. - Navigate to OTTO SEO in the left sidebar, then open SEO Automation (URL: /seo-automation-v3). - Open your connected website and select OTTO Settings. - Scroll to the Crawler IP Addresses section to copy the current IP list directly from the platform. Cloudflare 1. Log in to your Cloudflare dashboard and select the affected domain. 2. Go to Security → WAF → Tools. 3. Under IP Access Rules, enter each Search Atlas IP address and set the action to Allow. 4. Optionally, create a WAF Custom Rule that matches the Search Atlas user-agent string and sets the action to Skip all security rules. Wordfence (WordPress) 1. In your WordPress admin, go to Wordfence → Firewall → Blocking. 2. Select Whitelist IPs that bypass all rules. 3. Add each Search Atlas crawler IP address and save. Generic WAF or Hosting Firewall 1. Locate the IP allowlist or whitelist section in your WAF or hosting control panel. 2. Add each Search Atlas crawler IP as a trusted source. 3. If your platform supports user-agent rules, also add the Search Atlas crawler user-agent string available in your OTTO Settings panel. robots.txt 1. Open your robots.txt file for editing. 2. Add a block explicitly allowing the Search Atlas crawler, using the exact user-agent string shown in your OTTO Settings panel, followed by Allow: /. 3. Make sure no subsequent Disallow rule overrides this permission. 4. Save and re-publish the file, then allow up to 15 minutes for the change to propagate. 🔄 Step 3 — Restart the OTTO Crawl 1. Go to OTTO SEO in the left sidebar, then open Site Audit → Overview (Website Overview). 2. Select your website to view the Website Overview page. 3. Click Recrawl Site to trigger a fresh crawl now that the IP addresses are whitelisted. ⏱️ Expected Crawl Completion Timelines Crawl completion time varies with site size and how quickly the whitelisting change propagates — smaller sites generally complete faster than very large sites, and OTTO may continue processing large sites in the background after the initial pass. If a crawl seems to be taking unusually long relative to your site's size, contact support so we can check on its progress. You can monitor real-time crawl progress from the OTTO SEO V3 dashboard. The page count will increment visibly once the block is resolved. 🚫 Common Mistakes to Avoid - Whitelisting the IP in only one layer when multiple security tools are active — check every layer (CDN, WAF, host firewall, plugin). - Forgetting to save or publish the whitelist changes before restarting the crawl. - Confusing the Search Atlas crawler with Google bot and applying rules to the wrong user-agent. - Not restarting the crawl after whitelisting — OTTO will not automatically retry until a new crawl is initiated. 💬 Still Need Help? If you need further assistance, open the chat widget in the bottom-right corner of the platform and type human teammate to be connected with a member of our team.

🕷️ OTTO SEO Crawl Volume and Schedule Explained

🔍 Overview If your site has a large number of pages, you may notice that OTTO SEO has crawled and indexed fewer pages than your total page count. This is expected behaviour. OTTO uses a smart crawling system designed to prioritise the pages most likely to impact your SEO performance, while keeping resource usage efficient for your site. 📊 Why OTTO Indexes a Subset of Your Pages OTTO does not crawl every page on your site in a single pass. Instead, it works through your pages progressively, applying the following logic: - Priority-based crawling: OTTO focuses first on your highest-value pages — typically those already receiving organic traffic, linked from your sitemap, or flagged as important in your site architecture. - Crawl budget allocation: Each connected site is assigned a crawl budget based on your Search Atlas plan. This budget controls how many pages OTTO processes per crawl cycle. - Incremental indexing: Pages are added to OTTO's index over multiple crawl cycles. A site with 65,000 pages will not be fully indexed after a single crawl — this is normal and by design. - Duplicate and low-value page filtering: OTTO automatically excludes near-duplicate pages, thin content pages, and URLs blocked by your robots.txt or noindex directives, which can reduce the indexed count compared to your total page count. In the example of a 65,000-page site showing 16,000 indexed pages, this reflects the progress of the current crawl cycle combined with priority filtering. The indexed count will grow with each subsequent crawl. 🕐 When Does OTTO Crawl Your Site? OTTO is designed to crawl during off-peak hours to minimise any impact on your site's performance and server load. Here is what you need to know about the crawl schedule: - Off-peak timing: Crawls are scheduled to run during low-traffic periods, typically overnight based on the time zone associated with your Search Atlas account. - Automatic scheduling: You do not need to manually trigger crawls. OTTO manages the schedule automatically once it is activated for your site. - Recurring cycles: OTTO re-crawls your site on a regular basis to detect new pages, content changes, and resolve previously identified issues. The frequency depends on your plan tier. - On-demand crawl triggers: If you publish a large batch of new pages or make significant structural changes, you can manually request a fresh crawl from within the OTTO dashboard to accelerate indexing. 🛠️ How to Check Crawl Status and Progress 1. Log in to Search Atlas and navigate to OTTO SEO → All Sites (SEO Automation) in the left sidebar, or go directly to /seo-automation-v3. 2. Select the site you want to review from the site selector at the top of the page. 3. Review the crawl summary panel, which shows the total pages discovered, pages indexed, and the timestamp of the last completed crawl. 4. Check the Crawl Progress indicator to see how far through the current cycle OTTO has progressed. 5. If you have recently added new pages and want OTTO to pick them up sooner, use the Trigger Crawl option if available on your plan. 💡 Tips to Improve Crawl Coverage - Submit an updated XML sitemap: Ensure your sitemap is connected to OTTO and reflects your full page inventory. OTTO uses the sitemap as a primary discovery source. - Audit your robots.txt: Confirm that important pages are not unintentionally blocked from crawling. - Remove noindex tags from pages you want indexed: OTTO respects noindex directives and will exclude those pages from its count. - Reduce redirect chains: Long redirect chains slow down crawling and can cause pages to be skipped within a given cycle. - Upgrade your plan for a higher crawl budget: If your site grows significantly, a higher-tier plan increases the number of pages OTTO can process per cycle. ❓ Frequently Asked Questions Will OTTO eventually index all 65,000 pages? Yes — over multiple crawl cycles OTTO will work through your full page inventory, subject to your plan's crawl budget. Pages that are accessible, indexable, and not filtered as low-value will be added incrementally. Does crawling affect my site speed? OTTO's off-peak scheduling and rate-limited crawling are specifically designed to avoid impacting your site's performance or server response times. Why did the indexed page count drop? A drop in indexed pages usually means OTTO detected that some pages became noindexed, were removed, returned errors (such as 404s), or were filtered out as duplicates during the latest crawl cycle. 🙋 Still Need Help? If you need further assistance, open the chat widget in the bottom-right corner of the platform and type human teammate to be connected with a member of our team.

⏱️ How Long Google Takes to Index OTTO Updates

🔍 What Happens After OTTO Deploys Your Changes When OTTO deploys updated meta titles or meta descriptions to your site, the changes are applied to your pages immediately. However, there are two separate timelines to be aware of: when the updates appear in Search Atlas Page Explorer and when they appear in Google Search results. 🗓️ How Long Does Google Indexing Take After your meta title or meta description is updated on your site, Google must crawl and re-index the affected pages before the new content appears in search results. This process is outside of Search Atlas's control and depends entirely on Google's crawl schedule. - Typical range: Most pages are re-crawled and updated in Google within 3 to 14 days. - High-authority or frequently updated sites may see changes reflected within a few days. - Lower-traffic or newer sites may take up to several weeks. - Google does not guarantee a specific crawl frequency for any page. This delay is normal and expected. It does not mean your changes failed to deploy. 📊 When Will Page Explorer Reflect the Updates Page Explorer in Search Atlas pulls live on-page data from your site. Once OTTO has successfully deployed a change, Page Explorer should reflect the updated meta title or description within 24 to 48 hours after the next scheduled data refresh for your project. If you want to verify the change was applied correctly right away, you can: 1. Visit the page directly in your browser and inspect the page source (Ctrl+U or Cmd+U). 2. Search for or the tag to confirm the new content is live. 🚀 How to Speed Up Google Re-Indexing While you cannot force Google to crawl immediately, you can request re-indexing through Google Search Console to increase the chance of a faster crawl: 1. Go to Google Search Console and select your property. 2. Paste the updated page URL into the search bar at the top and press Enter. 3. Click Request Indexing in the URL Inspection panel. 4. Google will add the page to its priority crawl queue. Results typically appear within a few days. This method works best for a small number of high-priority pages. For large-scale deployments, allow the standard 3–14 day window. ✅ How to Confirm OTTO Deployed Successfully Before waiting on Google, first confirm that OTTO applied your changes correctly: - Navigate to OTTO SEO in the left sidebar of Search Atlas, then open SEO Automation. - Locate the relevant page and check the deployment status. A green deployed indicator confirms the update was pushed successfully. - If the status shows an error or is still pending, review the OTTO deployment log for that page and re-deploy if needed. 💡 Key Takeaways - OTTO applies meta changes to your site immediately upon deployment. - Page Explorer reflects updates within 24–48 hours of the next data refresh. - Google reflects updates in search results within 3–14 days on average. - Use Google Search Console URL Inspection to request faster re-indexing for priority pages. - A delay in Google does not mean the deployment failed — always verify in your page source first. If you need further assistance, open the chat widget in the bottom-right corner of the platform and type human teammate to be connected with a member of our team.

⏳ OTTO Extended Crawl Times — Diagnosis and Communication

🔍 What Is a Normal OTTO Crawl Time? Most OTTO crawls complete within 15–45 minutes depending on site size and server response speed. Crawls on larger sites (5,000+ pages) or sites with slow hosting may take up to 60–75 minutes. A crawl running beyond 90 minutes is considered extended and requires escalation. ⚠️ User-Actionable Causes vs. Backend System Status Before escalating, it helps to understand whether the delay is something you can influence or whether it requires backend investigation. User-actionable causes (you can address these now): - The site has a very large number of pages — OTTO crawls every accessible URL, so sites with 10,000+ pages naturally take longer. - Slow server response times on your website — if your hosting is under load, OTTO receives pages more slowly. - A previous crawl cycle was incomplete — starting a new crawl before the last one finishes can queue delays. - Robots.txt or crawl-rate directives are throttling OTTO — check that your site does not restrict crawl frequency. Backend system causes (require Support investigation): - The crawl job is stuck in a processing queue behind other jobs. - A backend worker has stalled mid-crawl. - A system-side timeout has not yet triggered a retry. - OTTO shows an active crawl status but no new pages are being processed. Only a Support teammate with backend access can distinguish between these two categories with certainty. 🗺️ How to Check Your Current Crawl Status in the Platform 1. Go to OTTO SEO in the left sidebar, then open SEO Automation (URL: /seo-automation-v3). 2. Open the OTTO dashboard for the relevant website. 3. Look at the crawl progress indicator at the top of the page — it displays pages crawled, pages remaining, and an estimated completion time when available. 4. If the progress bar has not moved for more than 20 consecutive minutes, the crawl may be stalled rather than simply slow. 5. Note the exact time the crawl started (visible in the crawl history log below the progress bar) so you can report the duration accurately if you need to escalate. Important: The platform progress indicator reflects the last-synced status. It may lag a few minutes behind actual backend processing. A crawl that appears frozen for under 20 minutes is likely still running normally. 📋 When to Provide a Time Estimate vs. Generic Guidance Use the table below to decide the right response at each stage. - Crawl running 0–60 minutes: No action needed. Share the normal range (15–45 min for small sites, up to 75 min for large sites) and advise the customer to wait. - Crawl running 60–90 minutes with visible progress: Reassure the customer that large or complex sites can take this long. Confirm progress is still moving and give a rough estimate based on pages remaining. - Crawl running 60–90 minutes with no visible progress for 20+ min: Escalate immediately — do not offer further generic troubleshooting. - Crawl running 90+ minutes regardless of progress: Escalate immediately so a teammate can check backend queue position and provide an accurate time estimate or restart the job. Only a Support teammate with backend access can provide a specific remaining-time estimate (for example, "35 minutes remaining"). If you cannot see this in the platform UI, do not guess — escalate instead. 🚨 Escalation Criteria and Steps Escalate to the Support team when any one of the following is true: - The crawl has been running for 90 minutes or longer. - The progress bar has shown no movement for 20 or more consecutive minutes at any point during the crawl. - The crawl status shows active but the pages-crawled counter has not increased in over 15 minutes. - OTTO displays an error message or a spinning loader with no status text. When you escalate, share the following information to speed up the investigation: 1. The domain being crawled. 2. The exact time the crawl started. 3. The current pages-crawled count and total pages shown in the UI. 4. Whether the progress bar has been moving recently or appears frozen. 5. Any error messages visible on the OTTO dashboard. 💡 Tips to Prevent Extended Crawl Times - Avoid starting a new crawl while a previous one is still active — wait for full completion before re-triggering. - Ensure your website's server is not under heavy load when running an OTTO crawl (avoid scheduling during peak traffic hours). - Check that your robots.txt does not include a low Crawl-delay value that forces OTTO to slow down. - For very large sites, expect the first crawl to take the longest — subsequent crawls are typically faster because OTTO only re-processes changed pages. 🙋 Need Help Right Now? If your crawl has been running for 90+ minutes or appears stalled and you need an accurate status check or time estimate, please escalate directly to the team. If you need further assistance, open the chat widget in the bottom-right corner of the platform and type human teammate to be connected with a member of our team.

🛠️ Fix OTTO Loading Errors and Audit Delays

🔍 Understanding the Issue If OTTO fails to load or your site audit appears stuck, you are not alone. These issues are usually caused by one of a few common factors: an incomplete site audit, browser-related conflicts, or a temporary sync issue within the platform. This article walks you through the most effective fixes so you can get OTTO running as quickly as possible. ⚙️ Step 1 — Check Your Site Audit Status OTTO requires a completed site audit before it can fully load and activate. If your audit is still in progress or has not been triggered, OTTO will not initialise correctly. 1. In the left sidebar, click OTTO SEO, then open Site Audit and select All Audits. 2. Check whether a site audit is shown as In Progress or Pending. 3. If no audit has been run yet, click Run Site Audit and wait for it to complete fully before attempting to load OTTO again. 4. If an audit is already in progress, allow it to finish — audit times vary depending on site size, but most complete within 5 to 30 minutes. Important: Do not refresh the page repeatedly while an audit is running. Excessive refreshing can interrupt the process and cause further delays. 🌐 Step 2 — Clear Your Browser Cache and Cookies Cached data and outdated cookies are a frequent cause of loading errors inside Search Atlas. Clearing them takes less than two minutes and resolves the majority of OTTO loading issues. 1. Open your browser settings (usually accessible via the menu in the top-right corner). 2. Navigate to Privacy and Security or Clear Browsing Data. 3. Select Cached images and files and Cookies and other site data. 4. Set the time range to All time and confirm the deletion. 5. Close and reopen your browser, then log back into Search Atlas. 6. Return to OTTO SEO in the left sidebar, open Site Audit, and select All Audits to attempt to load OTTO again. 💡 Step 3 — Try a Different Browser or Incognito Mode Browser extensions such as ad blockers, VPNs, or script managers can interfere with how OTTO loads. Testing in a clean environment helps identify whether the issue is browser-specific. - Open a new Incognito or Private window in your current browser and log in to Search Atlas. - Alternatively, try a completely different browser (for example, switch from Chrome to Firefox or Edge). - Navigate to OTTO SEO in the left sidebar, open Site Audit, and select All Audits to check whether OTTO loads correctly. If OTTO loads successfully in incognito mode or a different browser, the issue is likely caused by an extension or a corrupted browser profile. Disable your extensions one by one to identify the culprit. 🔄 Step 4 — Refresh Your OTTO Connection Sometimes a simple session reset is all that is needed to restore the connection between OTTO and your site data. 1. Log out of Search Atlas completely using the account menu. 2. Wait 30 seconds, then log back in. 3. Go to OTTO SEO in the left sidebar, open Site Audit, and select All Audits. 4. If prompted to reconnect or reactivate OTTO, follow the on-screen instructions to complete the setup. 📊 Step 5 — Verify Your Site Is Properly Connected OTTO can only function correctly when your site is connected and verified within the platform. A missing or broken site connection will prevent OTTO from loading. 1. In the left sidebar, click Site Metrics (Site Explorer) to go to /site-explorer/list. 2. Confirm that your website appears in the list and shows an Active status. 3. If your site is missing or shows an error, re-add it by clicking Add New Site and completing the verification steps. 4. Once the site is confirmed as active, return to OTTO SEO in the left sidebar, open Site Audit, select All Audits, and try loading OTTO again. ✅ Quick-Reference Checklist - Site audit is fully completed — not in progress or pending. - Browser cache and cookies have been cleared. - OTTO loads correctly in incognito mode or a different browser. - You have logged out and back in to refresh your session. - Your site is active and verified under Site Metrics (Site Explorer). 🆘 Still Experiencing the Error? If you have worked through all of the steps above and OTTO is still not loading, our team is ready to investigate further. If you need further assistance, open the chat widget in the bottom-right corner of the platform and type human teammate to be connected with a member of our team. When you reach out, please have the following information ready to help us resolve your issue faster: your website URL, the browser and version you are using, the exact error message displayed (a screenshot is helpful), and the current status of your site audit. Additional Notes Navigate directly to https://dashboard.searchatlas.com/site-audit/list to check audit status. Audit runs automatically when you connect a project to OTTO; no action required while it runs. Large sites can take the full 24 hours.

🤖 Exclude Pages from OTTO Crawl Settings

🧭 Overview OTTO, Search Atlas's OTTO SEO automation engine, crawls your site to discover and apply on-page optimizations. By default, OTTO will process all pages it can reach within your project. If you want to prevent OTTO from crawling or optimizing certain pages — such as Spanish-language pages, staging URLs, or low-priority sections — you do this through Crawl Settings inside Search Atlas, not through robots.txt or noindex tags. This article explains how to set up OTTO-specific exclusions, clarifies whether OTTO changes are permanently written to your site, and covers how OTTO's reporting and integration features work. ⚠️ OTTO Exclusions vs. robots.txt — Key Difference A common point of confusion is the difference between site-wide crawl rules and OTTO-specific exclusions. - robots.txt / noindex tags — These are server-level directives that instruct all crawlers (Google, Bing, OTTO, etc.) how to behave. Editing robots.txt does not exclusively affect OTTO and is not the correct method for managing what OTTO processes. - OTTO Crawl Settings — These are exclusion rules configured entirely within Search Atlas. They tell OTTO which URLs to skip during its crawl and optimization passes, without touching your site's server configuration or affecting other crawlers. Always use OTTO Crawl Settings when you want to control what OTTO specifically crawls or optimizes. 🚫 How to Exclude Pages from OTTO Crawl To exclude specific pages, directories, or URL patterns from OTTO's crawl, you will need to configure exclusion rules inside OTTO's Crawl Settings for your project. Because the exact navigation path to these settings may vary depending on your account setup, we recommend reaching out to our support team so they can guide you directly to the correct location and confirm the right exclusion pattern for your use case (for example, excluding all Spanish-language pages via a language subdirectory pattern). When your exclusion rules are saved, OTTO will skip all matching URLs during its next crawl cycle. Any optimizations OTTO had already applied to those pages before the exclusion was added will remain in place unless you manually revert them. 🔄 Dynamic vs. Permanent Changes A frequent question is whether OTTO's optimizations are written permanently to your site. - Dynamic changes — Some OTTO optimizations are applied dynamically and are managed through Search Atlas. If you disconnect OTTO or remove a rule, those changes can be rolled back without a permanent edit to your CMS or server files. - Permanent changes — Other optimizations (such as those pushed directly to your CMS) are written to your site and persist independently of OTTO. These require manual intervention to revert. If you are unsure whether a specific OTTO action is dynamic or permanent, contact our support team before making changes. 📊 Reporting and Integrations OTTO provides reporting on the optimizations it has applied across your project. You can review which pages have been processed, which changes are active, and track the impact of OTTO's work over time. OTTO also integrates with your CMS, allowing certain optimizations to be pushed directly to your site from within Search Atlas. If you need further assistance, open the chat widget in the bottom-right corner of the platform and type human teammate to be connected with a member of our team.

🛠️ Website Studio Sitemap & Noindex Bug Fixes

🔍 Overview Some Website Studio websites experienced technical SEO bugs that affected sitemap accuracy, noindex tag behaviour, page indexability, and overall SEO scores. The issues described in this article have been resolved and deployed to production. This article explains what happened, what was fixed, and what you should do now to verify your site is healthy. 🐛 What Bugs Were Present? The following confirmed bugs affected Website Studio sites: - Homepage missing from sitemap: The XML sitemap omitted the homepage URL, reducing crawl coverage. - Incorrect noindex tags on published pages: Valid, publishable pages were incorrectly marked as noindex, preventing Google from indexing them and causing ranking drops. - Title tags rewritten on the SERP: A related bug caused title tags to be overridden in search results, further contributing to ranking loss. - Preview and draft domains were indexable: Unpublished draft and preview versions of sites were publicly crawlable and could appear in Google search results, competing directly with your live site. - Low SEO scores on Desktop and Mobile: Generated websites were receiving unexpectedly low SEO scores due to a scoring calculation error. - URL pollution from editor and preview URLs: Legacy editor, preview, and duplicate URLs were leaking into the live site and its sitemap, inflating URL counts and confusing crawlers. ✅ What Has Been Fixed? All of the issues listed above have been resolved and are fully deployed to production. Here is a summary of what was corrected: - The sitemap generator now correctly includes the homepage and excludes all noindex, draft, and preview URLs. - Noindex tags are no longer applied incorrectly to published, indexable pages. - Title tags are preserved as set in Website Studio and are no longer overridden. - Preview and draft domains are now blocked from indexing. - SEO scores for generated websites now reflect accurate Desktop and Mobile evaluations. - Editor, preview, and duplicate URLs are no longer injected into the live site or its sitemap. 📋 Steps to Verify Your Site After the Fix Even though the bugs are resolved, we recommend taking the following steps to confirm your site is in good shape and to help Google re-crawl any affected pages promptly. 1. Open Website Studio: Navigate to Website Studio from your platform dashboard. 2. Check your published site settings: Review your pages to confirm no pages are still marked noindex unintentionally. 3. Inspect your sitemap: Visit your sitemap URL and confirm the homepage and all key published pages are present, and that no preview or draft URLs appear. 4. Check Google Search Console: If you have Google Search Console connected, review your Coverage and Sitemaps reports for any lingering indexing errors or excluded pages. 5. Request re-indexing if needed: For any pages that were previously excluded incorrectly, use the URL Inspection tool in Google Search Console to request re-indexing. If you need further assistance, open the chat widget in the bottom-right corner of the platform and type human teammate to be connected with a member of our team.

🔍 Find Broken Links in Your OTTO SEO Audit

🗺️ Overview When Search Atlas runs an OTTO SEO audit on your site, it scans every crawlable URL and flags links that return error responses. If your audit has flagged dead links — for example, 157 broken URLs — this article walks you through exactly where to find them and how to review the full list. 🧭 How to Access the OTTO SEO Audit 1. Log in to your Search Atlas account. 2. In the left sidebar, click OTTO SEO, then All Sites (SEO Automation). 3. Your most recent audit results load automatically on the OTTO SEO Overview page (/seo-automation-v3). 🔗 How to Find the Broken Links List 1. On the OTTO SEO Overview page, scroll down past the summary score cards until you reach the audit pillars (Technical, On-Page, Content, etc.). 2. Locate the Technical pillar section. 3. Inside the Technical pillar, look for the Issues with Links section. This is where all broken link findings are grouped. 4. Click on the Issues with Links heading or its expand arrow to open the full list of flagged URLs. 5. You will see each broken link listed with its source page (the page containing the bad link), the broken URL itself, and the HTTP error code returned. 📋 Understanding the Broken Links Results Each entry in the broken links list tells you: - Source page: The page on your site where the broken link appears. - Broken URL: The destination URL that is returning an error. - Error type: The HTTP status code detected (such as 404 Not Found, 410 Gone, 403 Forbidden, or 5xx server errors). The audit checks for a range of error responses — not just 404s — so you may see several different error types in your list. ✅ How to Fix Broken Links Once you have your full list, here are the recommended steps to resolve broken links: 1. Remove the link if the destination page no longer exists and has no replacement. 2. Update the link to point to the correct, live URL if the page has moved. 3. Set up a redirect (301) on your server if the old URL should forward visitors to a new page. 4. After making fixes, return to OTTO SEO and re-run the audit to confirm the broken links have been resolved. ⚠️ Things to Keep in Mind - The number shown on the overview card (for example, 157 dead links) reflects all broken links detected across your entire site at the time the audit was last run. - AI-suggested replacement URLs in the Issues with Links section are generated to guide you — always review suggestions before applying them to confirm they point to the most relevant page on your own domain. - If the Issues with Links section appears blank or the page does not load correctly when a large number of broken links is present, try refreshing the page. This is a known display issue that is actively being resolved. - To get the most up-to-date broken link data, trigger a fresh audit after you have completed fixes. 💬 Need More Help? If you need further assistance, open the chat widget in the bottom-right corner of the platform and type human teammate to be connected with a member of our team.

OTTO Showing Stale Data After On-Page Fixes

🧭 Overview After deploying on-page fixes through OTTO, you may notice that the platform still shows old data — outdated meta tags, missing blog posts, an incorrect site icon, or SEO scores that haven't improved. This is almost always caused by a slow-loading website preventing Search Atlas's crawler from fully scanning your pages, not a sync issue with your CMS or content tool. This article explains what causes stale OTTO data, how to confirm it is a crawl speed problem, and the steps to resolve it. ⚠️ Why This Happens Search Atlas crawls your website to read live on-page data and verify that deployed fixes are in place. If your site loads slowly, the crawler may time out before it can retrieve a page's content. When this happens: - OTTO continues to display the last successfully crawled version of your data. - Fixes you have deployed appear undetected, even though they are live on your site. - Elements like blog posts, site icons, and meta descriptions may not match what a visitor sees in a browser. This symptom is commonly mistaken for a CMS sync gap or a content ingestion error, but the root cause is page performance, not a platform configuration problem. 🔍 How to Confirm a Crawl Speed Issue Before troubleshooting, confirm that slow site performance is the likely cause by checking the following: 1. Test your page speed. Open Google PageSpeed Insights (pagespeed.web.dev) and enter your domain. A score below 50 on mobile or desktop, or a Time to First Byte (TTFB) above 600 ms, indicates your site is slow enough to affect crawlers. 2. Check your hosting server response. Use a tool such as GTmetrix or Pingdom to measure server response time. Anything above 1–2 seconds can cause crawler timeouts on content-heavy pages. 3. Confirm fixes are live in a browser. Visit the affected pages directly in an incognito browser window and verify your deployed changes are visible. If they appear correctly in a browser but not in OTTO, the issue is crawl-side, not deployment-side. 🛠️ Steps to Resolve the Issue 1. Improve your site's load time. Work with your developer or hosting provider to reduce page load time. Common fixes include enabling caching, compressing images, using a CDN, and minimising render-blocking scripts. The goal is a TTFB under 400 ms and a full page load under 3 seconds. 2. Allow time for re-crawling. Once performance improvements are in place, Search Atlas will re-crawl your site on its next scheduled cycle. After the crawl completes, OTTO should reflect your deployed fixes accurately. 3. Contact support if data remains stale. If your site speed has improved and OTTO still shows outdated data after a reasonable period, reach out to the Search Atlas support team so they can investigate the crawl status for your domain directly. If you need further assistance, please contact our support team via the chat widget in your Search Atlas dashboard.