Home πŸ” Site Audit & Technical SEO

πŸ” Site Audit & Technical SEO

Camilo Aponte
By Camilo Aponte
β€’ 24 articles

πŸ” Diagnosing Crawled Not Indexed Pages

πŸ“Œ What "Crawled – Not Indexed" Means When a page is crawled but not indexed, Google has visited the page but decided not to add it to its search results. The page is not blocked and Google can reach it. Google simply did not consider it worth storing yet. This status is common, especially on new sites, large sites, and sites with many similar pages. It does not always mean something is broken, but it is a signal that the page can be improved. ❓ Is There a One-Click Tool for a Single URL? Customers often ask whether they can give Search Atlas a URL and get back a list of exactly what to fix. There is no single tool that does this, because Google does not tell anyone the exact reason a page was left out of its index. Instead, you can combine two tools to find the most likely causes: - Site Audit checks the technical side: tags, redirects, links, speed, and crawl settings. - Content Genius checks the content side: depth, relevance, and how your page compares with pages that already rank. Used together, they cover the two main reasons pages stay out of the index: technical signals and content quality. βœ… Step 1: Confirm the Page Status Before you change anything, confirm the page is really crawled but not indexed. 1. Open Google Search Console and go to the URL Inspection tool. 2. Paste the full URL of the page. 3. Check the status. Look for wording such as "Crawled – currently not indexed." 4. Make sure you are testing the exact version of the URL you want indexed (with or without www, http or https, trailing slash). If the status says something different, such as "Discovered – currently not indexed" or "Excluded by noindex tag," the fix will be different. The steps below still help with most cases. πŸ› οΈ Step 2: Check Technical Issues with Site Audit Run a Site Audit on the domain, or open your most recent audit if you already have one. Then look for issues that affect the affected URLs. - Noindex tags: a noindex directive tells Google to leave the page out. Remove it if the page should be indexed. - Canonical tags: if the canonical points to a different URL, Google may treat your page as a duplicate and skip it. - Robots.txt rules: make sure the page and its important resources are not blocked. - Status codes and redirects: pages should return a normal 200 status. Redirect chains, 404s, and soft errors can cause problems. - Duplicate or missing titles and meta descriptions: many pages with the same titles look like duplicates. - Orphan or weakly linked pages: pages with few or no internal links look unimportant to Google. - Page speed and loading problems: very slow pages or pages that fail to load content can be skipped. - Sitemap issues: confirm the page is listed in your XML sitemap and the sitemap has no errors. Fix the issues that apply to your page first. These are usually the quickest wins. ✍️ Step 3: Improve Content Quality with Content Genius If Site Audit shows no technical problems, content quality is the most likely cause. Google often skips pages that are thin, repetitive, or similar to pages already in its index. 1. Open Content Genius and enter the main keyword or topic the page targets. 2. Compare your page with the top-ranking results for that topic. 3. Look for gaps such as missing subtopics, short sections, or questions that you do not answer. 4. Expand the page with useful, original information. Add examples, data, images, or clear step-by-step guidance where it makes sense. 5. Improve headings so the structure is easy to scan. 6. Avoid copying text from other pages, including your own. Each page should have a clear, unique purpose. Aim for a page that gives readers something they cannot get from the other pages on your site or from the top results. πŸ”— Step 4: Strengthen Internal Links Internal links tell Google which pages matter. For each page that is not indexed: - Add links to it from related, already indexed pages. - Use clear, descriptive link text instead of "click here." - Link from high-traffic pages, such as your homepage, category pages, or popular articles, when it is relevant. - Make sure the page is not more than a few clicks from your homepage. πŸš€ Step 5: Request Indexing and Wait 1. After you make your fixes, go back to URL Inspection in Google Search Console. 2. Click Request Indexing for the updated page. 3. Run a new Site Audit to confirm the technical issues are resolved. 4. Wait one to four weeks, then check the status again. Requesting indexing does not guarantee results. Google decides based on the quality and value of the page, so the improvements in Steps 2 to 4 matter most. πŸ’‘ Common Causes at a Glance - Thin content: use Content Genius to add depth and unique value. - Duplicate or near-duplicate pages: merge them or set the correct canonical tag. - Noindex or wrong canonical: find it in Site Audit and correct it. - Weak internal linking: add links from relevant pages. - New site or low authority: keep publishing quality content and be patient. - Too many low-value pages: remove or consolidate pages that do not serve a purpose. πŸ™‹ Frequently Asked Questions Can I hand Search Atlas a URL and get a full fix list? Not in a single step. Use Site Audit to find technical issues and Content Genius to improve content. Together they point you to the most likely fixes. How long does indexing take after I fix a page? It can take from a few days to several weeks. Larger or newer sites often wait longer. Should I delete pages that stay unindexed? Only if they have no real purpose. Otherwise, improve them, add internal links, or combine them with a stronger page. Does a crawled-not-indexed page hurt my whole site? A few of them are normal. A large share of low-quality pages can reduce how Google views your site overall, so it is worth cleaning them up. **πŸ’¬ Need More Help? If you need further assistance, open the chat widget in the bottom-right corner of the platform and send your message. A member of our support team will reply in the chat.

Last updated on Nov 09, 2026

πŸ” Site Audit REST API Contract and Quotas

πŸ“Œ What this article covers This guide explains the Site Audit REST API for authenticated Search Atlas accounts: starting a crawl, how crawl_budget is applied, where to read status, settings, and URL issues, and how to recrawl a domain without changing settings or creating an OTTO project. It also covers permissions, quota, archive slots, rate limits, response fields, and subdomains. πŸš€ Start an audit Use POST /api/v2/site-audit/ with your account API credentials. Send the domain you manage and the crawl settings for that run, including crawl_budget. The call creates or updates the Site Audit record only. It does not create an OTTO project, and you should not send OTTO project fields. A successful response returns the audit identifier, the domain, the accepted crawl budget, and a status such as queued or running. Save the identifier. You need it for later reads and for a recrawl. πŸ“Š Crawl budget on a new domain Yes. If you send crawl_budget: 25 for a new domain, that audit is capped at 25 URLs. The crawler does not expand to the whole site. A higher plan allowance does not override the value you sent. - The cap applies only when 25 is within your remaining crawl quota. - If fewer than 25 URLs of quota remain, the crawl stops when quota runs out. - Pages that cannot be fetched still count toward the attempt limit for that run. - Changing the budget later requires a new request. It does not rewrite a crawl already in progress. βš™οΈ Read status, settings, and issues After POST returns an audit id, use the authenticated GET reads for that same audit. The current paths are listed in the in-product API reference so you always call the version tied to your account. Use three separate reads: - Status returns state, progress, URLs crawled, and whether the run is queued, running, complete, or failed. - Settings returns the saved domain settings, including the crawl budget used for the latest run. - URL issues returns the issue list for crawled URLs, including severity and the affected URL. Do not poll a different domain or another account's audit id. Empty issues usually mean the crawl has not finished or no issues were found. πŸ› οΈ Recrawl without changing settings To recrawl an existing domain, call POST /api/v2/site-audit/ again for that same domain and audit. Omit settings you do not want to change, and do not send any OTTO project payload. Saved settings, including the existing crawl budget, stay as they are. 1. Confirm the domain already has a Site Audit id. 2. Send the recrawl request with that domain and audit id only. 3. Do not include new settings or an OTTO project. 4. Poll the status read until the run completes, then refresh URL issues. A recrawl replaces the previous run's results for that domain. It does not merge two crawls and it does not create a second project. πŸ’‘ Permissions, quotas, slots, and rate limits The API is for authenticated use by your Search Atlas account. It is not an open public endpoint. You may audit domains you add to your own account. Do not share tokens or audit a site you do not manage. - Quota: each URL attempted in a crawl counts toward your Site Audit crawl quota. Quota is consumed when the crawl runs, not when you merely save settings. - Archive and slots: a finished audit uses an active slot until you archive it. Archiving frees the slot. It does not refund URLs already crawled and it does not erase saved settings used for a later recrawl. - Rate limits: keep one active crawl per domain. Parallel crawls of the same domain can be rejected. If a call is throttled or quota is exhausted, wait and retry later instead of launching another crawl. πŸ” Response fields and subdomain isolation Create and status responses include the audit id, domain, status, accepted crawl budget, and URLs crawled. The settings read returns the stored settings. The issues read returns one entry per affected URL with the issue type and severity. Extra fields may appear as the API evolves. Ignore fields you do not use. A root domain and its subdomains are isolated. An audit of example.com does not crawl blog.example.com unless that host is included in the crawl scope or audited as its own domain. Each host has its own status, settings, issues, and quota usage. Archiving one host does not archive the other. **πŸ’¬ If you still need helpIf you need further assistance, open the chat widget in the bottom-right corner of the platform and send your message. A member of our support team will reply in the chat.

Last updated on Nov 08, 2026

πŸ—ΊοΈ Fix Sitemap Indexing Stuck at Zero

πŸ“Œ Why the indexed count stays at zero A healthy sitemap and a clean robots.txt only tell Google which pages exist and that crawling is allowed. They do not guarantee that pages are indexed. The indexed number in Google Search Console can stay at zero while Google is still discovering URLs, or when pages are crawled but excluded. Search Atlas cannot force Google to index a site. You can confirm the setup, resubmit the sitemap, and request indexing for important URLs. Google decides what to crawl and when. βœ… Check the basics first - Confirm the sitemap URL returns a valid XML file and lists the live page URLs you expect, including https and the correct domain. - Open robots.txt and confirm it does not block those URLs or the sitemap itself. - Spot-check a few listed URLs. Each should return a normal page, not a redirect loop, login wall, or error. - On those pages, confirm there is no noindex tag and that the canonical URL matches the URL in the sitemap. - In Search Atlas, open the correct site project and confirm Google Search Console is connected to the same property that owns the sitemap. A domain property and a URL-prefix property can show different data. πŸ”„ Resubmit the sitemap in Search Console 1. In Google Search Console, select the property that matches the live site. 2. Open Sitemaps in the left menu. 3. If the sitemap is already listed, note the status. Success with zero indexed pages is common on new or recently changed sites. 4. Enter the sitemap path, such as sitemap.xml, in the add field and choose Submit. Submitting again asks Google to read the file again. It is not an instant recrawl of every URL. 5. Return to Search Atlas and refresh or re-sync Search Console data for that project so the platform shows the latest status. πŸš€ Request indexing for key pages 1. In Google Search Console, open URL inspection at the top of the page. 2. Paste one important URL from the sitemap, such as the homepage or a priority service page, and press Enter. 3. Review the result. Look for blocked by robots.txt, excluded by noindex, duplicate or alternate page, or crawled but not indexed. 4. If the URL is eligible and the Request indexing button is available, select it. Use this for a small set of priority URLs, not every URL in the sitemap. 5. Repeat for a few representative URLs. If those are excluded, fix the reported reason before requesting more. πŸ› οΈ Use Search Atlas to find blockers - Run or open the latest Site Audit for the project and filter for indexability issues, blocked pages, canonical conflicts, and redirect or status-code problems. - Compare audit URLs with the sitemap. Pages missing from the audit, or listed with errors, will not move the indexed count. - If OTTO recommendations are waiting, review indexation and technical items and apply only the fixes you intend to publish. - After fixes go live, resubmit the sitemap and request indexing again for the corrected URLs. ⏳ What to expect after a recrawl request Google may take days or longer to recrawl and update the indexed count. Zero does not always mean the sitemap was rejected. Check the Pages or indexing report for discovered, crawled, and excluded URLs. A rising discovered count with a low indexed count usually means quality, duplication, or canonical selection, not a broken sitemap. If the sitemap status is an error, the property is wrong, or URL inspection shows a sitewide block after you have corrected the pages, If you need further assistance, open the chat widget in the bottom-right corner of the platform and send your message. A member of our support team will reply in the chat.

Last updated on Nov 08, 2026

πŸ” OTTO Slots Versus Site Auditor Quota

πŸ“Œ Why slots and quota are different Search Atlas uses more than one limit. Team seats control how many people can log in. OTTO slots control how many websites can use OTTO. Site Auditor quota controls how many websites you can add to Site Auditor. Unused seats or OTTO slots do not raise your Site Auditor quota. πŸ” Why an 11th site is blocked If you have 13 seats or OTTO slots but only 10 sites in the list, that extra capacity is not Site Auditor room. The message that you do not have quota is checking the Site Auditor project limit, not your seat count. A plan can allow 13 people or 13 OTTO sites while Site Auditor still stops at 10 projects. The visible list can also be shorter than the quota already in use: - Archived, paused, or removed projects may still count until they are fully deleted. - A site that did not finish setup can still hold a project. - A search box or status filter may hide some items. - Projects owned in the same plan by another workspace or teammate still count. βš™οΈ How to check each limit 1. Open your plan or usage area and note three numbers separately: team seats, OTTO website slots, and Site Auditor projects. 2. Open Site Auditor and clear search or status filters so hidden projects appear. 3. Count every project, including paused or incomplete ones. 4. Open OTTO and count only sites with OTTO turned on. That count uses OTTO slots, not auditor quota. 5. Compare the Site Auditor count with the Site Auditor limit before you add another website. πŸš€ How to free a Site Auditor spot 1. In Site Auditor, open a project you no longer need. 2. Delete the project completely. Pausing it may not free quota. 3. Refresh the project list and confirm the count dropped. 4. Add the new website again. To keep more than your current Site Auditor limit, upgrade the Site Auditor or plan quota. Adding seats or OTTO slots alone will not remove this error. πŸ’‘ If the numbers still do not match Note the three limits you see and how many projects remain after filters are cleared. If you need further assistance, open the chat widget in the bottom-right corner of the platform and send your message. A member of our support team will reply in the chat.

Last updated on Nov 08, 2026

πŸ” Shopify Pixel Check Stuck Processing

πŸ” What a stuck check means The pixel check looks at your live Shopify storefront for the Search Atlas snippet. If Check again does nothing, the check is usually still running, blocked by the browser, or never started because the page did not refresh. Seeing the code in your page source is a good sign, but the checker only counts the snippet on the published storefront URL saved in your project. βš™οΈ Confirm the snippet is live Open your store in a private browser window, not the Shopify theme editor preview. View the page source and search for the Search Atlas pixel. It should appear once, near the closing head tag, exactly as copied from Search Atlas. - Edit theme.liquid on the theme that is currently published, then save and exit the editor so Shopify publishes the change. - Do not place the snippet only in a draft theme, a section, or the checkout page. - Remove duplicate copies. One snippet is enough. - If the store has a password page, open the storefront with the password or temporarily disable it so the checker can read the real homepage. - Match the domain in Search Atlas to the live store URL, including www if that is what customers use. πŸ› οΈ Clear a check that will not click A check that looks frozen is often a disabled button waiting on a request that never finished. Reset it before you change the theme again. 1. Stay on the pixel page and wait two minutes. Do not click Check again repeatedly. 2. Hard refresh the Search Atlas page. On Windows use Ctrl+Shift+R. On Mac use Cmd+Shift+R. 3. If the button is still inactive, sign out, sign back in, and open the same project. 4. Try another browser with extensions disabled. Ad blockers and privacy tools can stop the check request. 5. Open your live store once in that same browser, then return to Search Atlas and run Check again a single time. πŸ“Š If the code is in the source but still fails Source code on your computer does not always match what our checker receives. Work through these storefront issues: - Cache: Shopify and your browser can serve an older theme. Wait 10 minutes, then recheck from a private window. - Wrong theme: Confirm the published theme is the one you edited. Theme previews are not checked. - Storefront apps: A page builder, geo redirect, or bot challenge can hide the homepage from automated checks. Pause the redirect for the homepage and try again. - Domain mismatch: A project set to a myshopify.com address will not pass if the snippet only loads on your custom domain, and the reverse is also true. - Edited snippet: Paste the pixel again from Search Atlas. Do not add extra quotes, comments, or Shopify liquid around it. πŸ’‘ After a successful check When the status changes from processing to connected, leave the snippet in theme.liquid on the published theme. Removing it later will stop data collection. If you switch themes, add the same snippet to the new published theme and run the check once more. πŸš€ Still stuck If the button stays inactive after a hard refresh and a clean browser, or the live source shows the exact snippet and the check still fails, contact support with your store URL and the theme name. If you need further assistance, open the chat widget in the bottom-right corner of the platform and send your message. A member of our support team will reply in the chat.

Last updated on Nov 08, 2026

πŸ“Š Download Site Issue Fix Reports

πŸ“‹ What the issue and fix report includes A site issue and fix report lists the technical problems found on a website and the recommended fix for each one. Use it to review suggested improvements before you decide whether to turn on OTTO. - Issue: the technical problem detected on the site. - Recommended fix: the change suggested to resolve that issue. - Site scope: results are shown per website, so each domain has its own report. You can review and download these suggestions without setting up or enabling OTTO. OTTO is optional automation. It is not required to see issues or export the report. ⬇️ Download the report without OTTO 1. Sign in to Search Atlas and open the project for the website you want to review, such as UGG Australia or Sheepskins Downunder. 2. Open the site’s Issues or Audit area. This is where technical issues and recommended fixes are listed. 3. Confirm the correct domain is selected so you do not export another site’s results. 4. Review the list. Each row should show the issue and its recommended fix. 5. Choose Download or Export and save the file. Repeat these steps for the second website. If a site has no issues listed yet, run or refresh the audit first, then return to the same area and download the report. Do not start OTTO setup only to unlock the download. πŸ’° Compare package cost with and without OTTO Pricing is shown inside the platform and depends on your plan, the number of websites, and whether OTTO is included. For two websites, compare both totals before you subscribe or change a plan. 1. Open Billing or the plan selector from your account menu. 2. Set the package to cover 2 websites. 3. View the price without OTTO. 4. Add OTTO, or switch to the OTTO-included option, and view the price with OTTO. 5. Confirm the billing period shown, such as monthly or annual, so both totals use the same term. Published checkout prices can change, so the amounts on your billing screen are the current package cost. The report download itself does not require you to purchase or activate OTTO. πŸ› οΈ If the download or price is missing - Refresh the page and confirm you are in the correct website project. - Check that the audit has finished. An in-progress audit may not have a full export yet. - Look for Download or Export near the issues list, not inside OTTO setup. - For pricing, use the in-platform plan comparison for exactly two websites, once with OTTO and once without it. **πŸ’¬ Get help in the platformIf you need further assistance, open the chat widget in the bottom-right corner of the platform and send your message. A member of our support team will reply in the chat.

Last updated on Nov 08, 2026

πŸ› οΈ Fix a Stalled Crawl and Firewall Whitelist

πŸ” Why analyze stats stay unchanged Analyze stats update only after Search Atlas finishes a fresh crawl of that project. If this is the only project that looks frozen, the crawl is usually stalled because the site, a firewall, a CDN, or a security plugin is blocking the crawler. Other projects keep updating because their hosts allow the bot through. A blocked crawl is not a data delay. The auditor never reaches the pages, so scores, issue counts, and page stats stay on the last successful run. ⚠️ Signs the crawl is blocked - Analyze stats do not change after you start a new audit. - The crawl stays queued, crawling, or failed for a long time. - Only one domain is affected while other projects update normally. - Your host, Cloudflare, a WAF, or a security plugin shows blocked bot requests. - The homepage loads in a browser but returns 401, 403, 429, or a challenge page to bots. πŸ›‘οΈ Whitelist the Search Atlas crawler Ask whoever manages hosting or security to allow the Search Atlas crawler before you run the audit again. Use the user agent and IP addresses shown in your project crawl or site audit settings. Do not guess addresses from old tickets. 1. Open the affected project and note the crawler user agent and IP list in the crawl or audit settings. 2. In your firewall, WAF, CDN, or security plugin, add those IP addresses to the allow list. 3. Allow the Search Atlas user agent. Do not require a CAPTCHA, JavaScript challenge, or login for that bot. 4. If you use Cloudflare or a similar CDN, set a skip or allow rule for those IPs and the user agent on Bot Fight Mode, rate limits, and country or hotlink blocks. 5. Allow both HTTP and HTTPS, and do not block common crawl paths such as robots.txt, sitemaps, or public pages. 6. Save the rule, then purge any security cache so the new allow list applies immediately. Whitelist only the addresses provided in the platform. If the list is not visible in the project, use live chat and ask a teammate for the current crawler details before changing production firewall rules. πŸš€ Trigger a fresh analyze run 1. Confirm the homepage and a few inner pages return a normal 200 status to the crawler, not a block or challenge page. 2. Open the stalled project and start a new site audit or crawl from the analyze area. 3. Leave the crawl running until it completes. Starting it repeatedly can keep it queued. 4. When the run finishes, refresh the project. Updated issue counts, page totals, and scores confirm the crawl succeeded. 5. If stats still match the old run, check host logs for the same block, adjust the allow rule, and run one more audit. πŸ“ Where to Find It UI path: Left sidebar β†’ OTTO SEO β†’ Site Audit β†’ Crawl Monitoring πŸ’‘ Checks that prevent another stall - Keep the allow rule active after plugin, CDN, or host updates. Those updates often reset security settings. - Avoid rate limits that treat a full-site crawl as an attack. - Make sure robots.txt is not disallowing the whole site. - Pause uptime or maintenance mode before the audit. - Compare this project with a working project. The difference is often one firewall rule on that domain only. **πŸ’¬ Still not updatingIf you need further assistance, open the chat widget in the bottom-right corner of the platform and send your message. A member of our support team will reply in the chat.

Last updated on Nov 08, 2026

βš™οΈ Site Audit quota blocks new OTTO sites

πŸ“Œ Why the quota message appears Adding a new OTTO site can fail with a quota exceeded message even when you still have unused seats and nothing is in Deep Freeze. Seats and OTTO site allocation are not the same as Site Audit capacity. An extra OTTO slot does not raise your Site Audit quota. The 10 limit customers see is the number of Site Audit slots on the account. Those slots are tracked separately from how many OTTO sites or seats you have. A site can still occupy a Site Audit slot after it looks unused in OTTO or Deep Freeze, which is why an 11th OTTO site can be blocked while 13 seats are available. πŸ› οΈ How to free a Site Audit slot To make room for a new site, both of the following must be done. Removing the site from OTTO alone is not enough. 1. Remove the site you no longer need from your OTTO SEO projects. 2. Also remove the Site Audit associated with that same site. The audit is a separate item and continues to use one of the 10 Site Audit slots until it is removed. After both removals, the Site Audit slot is free and the new site can be added to the account. πŸ’‘ What does not free the slot - Buying or assigning an extra OTTO slot does not increase the Site Audit limit. - Having more seats than the Site Audit limit does not raise that limit. Seats and Site Audit slots are counted separately. - An empty Deep Freeze does not mean a Site Audit slot is available. The associated Site Audit can still be using the quota. βœ… Before you try again Confirm the site you want to replace has been removed from your OTTO SEO projects and that its Site Audit has also been removed. Then add the new site. If the quota message continues after both items are gone, the remaining Site Audit slots may still be in use by other sites. If you need further assistance, open the chat widget in the bottom-right corner of the platform and send your message. A member of our support team will reply in the chat.

Last updated on Nov 08, 2026

πŸ”„ Refresh a Stale Project Crawl

πŸ“Š Why analyze stats look outdated If only one project is not showing updated analyze stats, its scheduled crawl may not have run. Smart Crawl runs every 7, 14, or 30 days, and the daily crawl job starts it when it is due. A firewall on the site can also block the crawler, so the project never gets fresh data even when a crawl is started. βš™οΈ Turn Smart Crawl off and on This is the first step support uses when a weekly crawl has not been triggered. 1. Open the affected project. 2. Turn Smart Crawl off. 3. Turn Smart Crawl back on. Turning Smart Crawl back on restarts its schedule and, for a single project, requests a crawl immediately. You can also use Site Audit's Recrawl Site button or OTTO's Manual reprocess button. A crawl that is already running, or a manual recrawl within the last 24 hours, will not start again; the next one stays on schedule. Use the chat widget if those options do not refresh the stats. πŸ› οΈ Whitelist crawler IPs after a firewall block If the crawl is still blocked, the site firewall is the usual cause. Ask the WordPress admin to check these protections: - Wordfence - Sucuri - The hosting provider firewall Whitelist every crawler IP listed at https://bots.searchatlas.com/ip_list.txt. Use the full current list, not a single address, because the crawler uses more than one IP. After the IPs are allowed, turn Smart Crawl off and on once more to request a new crawl. If a crawl is already running or one was manually started within 24 hours, it stays on the existing schedule instead. πŸ’¬ If stats still do not update If you need further assistance, open the chat widget in the bottom-right corner of the platform and send your message. A member of our support team will reply in the chat. Tell them which project is stale, that Smart Crawl was toggled, and whether the crawler IPs were whitelisted. Support can help confirm or start a recrawl and keep the case open until updated analyze stats show.

Last updated on Nov 04, 2026

πŸ“§ Site Audit CSV Export Not Received

πŸ“§ When the export email does not arrive If you requested an export from the Site Audit Issues page and the email with the download link has not arrived, there is no way to download that same file from Site Audit. Export on that page only starts another email, and the file it sends is an Excel workbook, not a CSV. πŸ› οΈ Export the data from Site Audit Open Site Audit for the site you were exporting. You can download other Site Audit files directly, but none of them is the Issues export the email was meant to deliver. That file is only available from the download link in the email. 1. Go to Site Audit for the website whose export you requested. 2. Use the export on the Site Audit page that has the data you need. On Overview, open Share and choose CSV export to download a crawled-page list. Page Explorer, a single page, and Page Pruning also download their own files. On the Issues page, Export only emails an Excel workbook, and that option is available on paid plans. 3. If you used a direct export, such as Overview CSV, Page Explorer, a single page, or Page Pruning, the browser downloads the file when it finishes and you can save it. If you used Export on the Issues page, nothing downloads. You get an email later with a link to an Excel workbook. Customers who did this confirmed they could get the export and no longer needed the missing email. πŸ’‘ What this does not require - If the missing email is for the Issues export, that email is how the file is delivered. A direct export from another Site Audit page does not replace it. - You can start a direct export from Overview, Page Explorer, a single page, or Page Pruning without asking support to check the missing email. If the missing email is for the Issues export, Site Audit cannot show you its status, so support is who checks that email. - A missing email does not stop you from downloading the separate Site Audit files, such as the Overview CSV, Page Explorer, a single page, or Page Pruning. It does stop you from getting the Issues export, because that Excel workbook is only delivered by the email. πŸ’¬ If you still need help If you need further assistance, open the chat widget in the bottom-right corner of the platform and send your message. A member of our support team will reply in the chat.

Last updated on Nov 04, 2026

πŸ” False sitemap and robots flags

⚠️ What these flags mean Site Auditor may report two related issues: Domain contains inaccessible sitemaps and Robots.txt references inaccessible sitemaps. These can appear on a new scan even when earlier scans of the same site read the sitemap without a problem. These messages mean the crawl did not successfully retrieve the sitemap. They do not always mean the file is broken on your site. βœ… Check that the files are publicly reachable Open your sitemap URL and your robots.txt URL directly in a browser. - The sitemap should load and list your URLs. - robots.txt should load and, if it declares a sitemap, that address should match the sitemap you expect. - If either file fails to load, returns an error, or is blocked for visitors, fix that on your site first, then ask support to recrawl. If both files open normally, the Site Auditor flags can still remain. In a confirmed case, support recrawled the account and the same findings stayed even though the URLs were accessible. πŸ› οΈ What support found The crawl requests sitemap URLs declared in robots.txt. A proxied fetch can return 429 or 403. A 403 is retried once without the proxy, but a 429 is not. If the sitemap is still not retrieved, Site Auditor treats it as inaccessible. A related score drop was fixed on our side. In that case, the customer confirmed the site score returned to above 900. The crawler now requests sitemap URLs named in robots.txt and retries a proxied 403 directly. A proxied 429 is not retried, so a sitemap can still be marked inaccessible. You do not need to change a healthy, publicly reachable sitemap or robots.txt to clear a flag caused by this crawler behavior. πŸ’¬ What to do next 1. Confirm the sitemap and robots.txt both open in a browser. 2. Note the scan date and the exact issue names shown in Site Auditor. 3. If the files are reachable and the flags remain, contact support so the account can be recrawled and, if needed, sent to engineering. If you need further assistance, open the chat widget in the bottom-right corner of the platform and send your message. A member of our support team will reply in the chat.

Last updated on Nov 04, 2026

πŸ“Š Why Content and UX Signals Show Zero

πŸ“Š What a zero score means A Content score or UX Signals score of zero means Search Atlas does not yet have enough usable data to grade that part of your site. It is not a penalty, and it does not mean your pages have no content or a broken experience. The score stays at zero until a completed crawl can measure the signals we track. Content signals look at how your website’s content covers important topics and keywords compared to your top competitors. UX Signals look at user interaction and real-world performance factors, including brand visibility and Core Web Vitals. If the crawl cannot see the pages, both scores can read zero even when the site looks fine in a browser. πŸ” Common reasons the scores stay at zero - The project is new. Scores appear only after the first successful crawl finishes. - No pages were crawled. A failed, blocked, or empty crawl leaves nothing to score. - The site is not connected correctly. A wrong domain, missing verification, or an incomplete setup can stop data collection. - The crawler is blocked. A robots.txt rule, firewall, login wall, or bot protection can hide the site from our crawler. - Important pages are excluded. Noindex tags, canonical issues, or crawl limits can leave the scored set empty. - The site relies on heavy scripting. If key content loads only after interaction, the crawl may record little or no content. - A refresh is still running. Older results can show zero until the latest crawl is processed. πŸ› οΈ Checks to run in the platform 1. Confirm the project domain. Make sure the property matches the live site, including https and www if you use them. 2. Open the latest crawl or audit. Check that it completed and that it lists pages, not an empty or failed run. 3. Review crawl access. Look for blocked URLs, robots.txt warnings, or authentication errors, then allow the Search Atlas crawler if your host is filtering bots. 4. Check a few key URLs. Confirm they are indexable, return a normal page, and are not stuck behind a login or cookie wall. 5. Start a fresh crawl. After you fix access, run the audit again and wait until processing finishes before reading the scores. 6. Compare with the page itself. Open the same URL in a browser. If content is missing in the crawl view but visible to you, the crawler is likely being blocked or the content is not in the initial HTML. πŸ’‘ What to expect after a successful crawl Once pages are crawled, Content and UX Signals should move off zero and reflect what was measured. A low score is different from zero: low means we scored the pages and found room to improve; zero means scoring has not started because data is missing. Large sites can take longer to process, so give the run time to finish before you judge the result. πŸš€ If the scores are still zero Gather the project name, the domain, the time of the last crawl, and any block or error message you see. If you need further assistance, open the chat widget in the bottom-right corner of the platform and send your message. A member of our support team will reply in the chat.

Last updated on Nov 05, 2026

⚠️ Fix False Site Suspension Warnings

⚠️ What this warning means Search Atlas may report that your website is suspended even when the site loads normally for visitors. This usually happens when a leftover cPanel account suspended page is still being served to our crawler. The live site is fine, but an old hosting file or cached response makes the tools treat the site as unavailable. The message does not mean Search Atlas has suspended your account or blocked your project. It only reflects what the crawler received from your domain. πŸ” Why a leftover cPanel page causes this Hosting panels such as cPanel can place a default suspended page on a domain when an account was paused, moved, or rebuilt. After the site is restored, that page is sometimes left in place and returned for some requests, user agents, or paths. Common signs include: - Your homepage looks normal in a regular browser, but tools still say the site is suspended. - A page titled Account Suspended, or text about contacting the hosting provider, appears for some visits. - The issue started after a host migration, DNS change, or a temporary hosting suspension that was later lifted. - Only certain URLs, or requests without cookies, still return the old page. πŸ› οΈ How to confirm the leftover page 1. Open your homepage in a private or incognito window and hard-refresh the page. 2. View the page source and search for words such as suspended, cPanel, or account has been suspended. 3. Check a few key URLs the same way, including the homepage with and without www, plus your main service or blog page. 4. Ask your hosting provider whether an old suspended document, redirect, or cache rule is still attached to the domain. 5. In Search Atlas, note which project or audit shows the suspension message so you can recheck that same URL after the host fixes it. If the crawler can still reach the leftover suspended page and stores it as the homepage, the warning is expected until your host stops serving that response. βœ… How to clear the false alert 1. Ask your host to delete or disable the leftover suspended page and any related redirect. 2. Confirm the domain now returns your real homepage for both www and non-www, and for http and https if both are in use. 3. Clear hosting, CDN, and server caches so old HTML is not served to crawlers. 4. Wait for DNS and cache changes to finish, then open the affected Site Audit project and click Recrawl Site. 5. Recheck the site status after the new crawl completes. The warning should clear once the crawler receives only your real pages and the old suspension URL is no longer recorded as the homepage. If it remains, the host is still serving that page. Do not delete your Search Atlas project just to clear the message. Fixing the response from the host is what resolves it. πŸ’¬ If the warning remains If your host confirms the suspended page is gone, caches are cleared, and a new crawl still reports the site as suspended, gather the affected URL and the approximate time of the last crawl. If you need further assistance, open the chat widget in the bottom-right corner of the platform and send your message. A member of our support team will reply in the chat.

Last updated on Nov 07, 2026

πŸ› οΈ Force Site Auditor Through Firewalls

πŸ” Why your crawl may be blocked Site Auditor needs to access your website to collect updated analysis data. A firewall, Web Application Firewall (WAF), bot-protection service, hosting security rule, or server restriction may block the crawler. When this happens, your project may continue showing older statistics or fail to start a new analysis. πŸ›‘οΈ Allow Site Auditor through your firewall 1. Open your firewall, WAF, CDN, or hosting security settings. 2. Review blocked requests and security events for the time of the Site Auditor crawl. 3. Allow legitimate Site Auditor crawler requests instead of blocking them as bots. 4. Reduce bot protection or challenge requirements for the crawl, if your security provider supports an exception. 5. Make sure your server allows requests from external crawlers and does not require a browser-only challenge, login, CAPTCHA, or JavaScript verification. 6. Confirm that your robots.txt file and server rules do not block the pages you want to analyze. Use the security provider’s verified crawler or request details when creating an allow rule. Avoid allowing all traffic or disabling your firewall globally. πŸš€ Trigger a fresh analysis 1. Sign in to Search Atlas and open the affected project. 2. Open Site Audit for the project. Crawl controls are Recrawl or Recrawl Site, and Crawl Settings. On a project card, Recrawl is also in the card menu. 3. Start a new crawl with Recrawl or Recrawl Site. If a crawl is already running, the button shows Crawling… or Recrawling and will not start another one. Crawl Settings changes the crawl configuration; it does not start the crawl. 4. Wait for the crawl to finish before checking the updated statistics. If the crawl does not begin, check whether the project has a crawl limit, an incomplete setup, or a website connection issue. Confirm that the project URL is correct and that the site is available publicly. πŸ“Š Check whether the crawl completed Review the crawl status and completion time in Site Auditor. Updated statistics may not appear until the crawl has fully finished. If the crawl completes but the numbers still look unchanged, compare the crawl date with the date of your latest website changes. For persistent issues, review your firewall logs while starting another crawl. A new blocked request can identify the rule that needs adjustment. πŸ’‘ Prevent future crawl interruptions - Keep a dedicated firewall exception for approved Site Auditor crawler traffic. - Monitor WAF rules after security policy changes or website migrations. - Check robots.txt and access rules whenever your domain, CDN, or hosting provider changes. - Do not require authentication or interactive verification on pages that Site Auditor must analyze. - Schedule crawls during a period when your server can handle the additional requests. If you need further assistance, open the chat widget in the bottom-right corner of the platform and send your message. A member of our support team will reply in the chat.

Last updated on Nov 09, 2026

πŸ› οΈ Troubleshoot Site Audit Issues Export Failures

πŸ” What an export failure means If your Site Audit issues export fails to generate, the requested file was not created successfully. This does not necessarily mean the crawl failed or that your audit results are unavailable. Check whether you can still view the issues inside Site Audit. πŸ› οΈ Check the audit and retry 1. Confirm the correct project. Open Site Audit for the affected domain and check that you are viewing the intended audit. 2. Check the crawl status on the Issues page. You can still request the export while a crawl is running, but the file reflects the results available at that time. If you want the latest crawl included, wait until processing finishes and the issues reload, then export. 3. Confirm that results are available. Open the issues view and check that the results load. 4. Refresh the page, reopen Issues, and click Export, then Start Export, once. If an export for the same audit and segment is already running, another click does not start a separate file. You are added to the email for the one already being built. 5. Check for a browser-specific problem. If the page behaves unexpectedly, try a private browsing window or another browser. Sign in and retry the same export. If the export has already failed repeatedly, including after a completed fresh crawl, stop retrying and report the failure. Another crawl is not a guaranteed fix for an export-generation problem. πŸ“‹ Gather details for support Include the following in your message so our team can investigate without asking you to repeat the same checks: - The affected domain and project name. - The date, approximate times, and time zone of the failed attempts. - How many times the export failed. - Whether the latest crawl completed and whether issues are visible in the platform. - Note the segment selected on the Issues page, if any. Search, Severity, Category, and Ignored filters change the on-screen list only and are not applied to the emailed Excel file. - The exact error message, or a screenshot of the failure. - Any troubleshooting steps you have already tried. For example: β€œThe issues export for [domain] failed five times today between 1:19 PM and 2:17 PM Philippine time, including after a completed fresh crawl. Issues are visible in Site Audit. Please investigate the export failure and confirm whether the file can be recovered or generated.” πŸ’¬ Request help or the export file If you need further assistance, open the chat widget in the bottom-right corner of the platform and send your message. A member of our support team will reply in the chat. If you need the file urgently, mention your deadline and ask whether the export can be recovered or generated. File availability and resolution time must be confirmed after investigation; an immediate replacement file cannot be guaranteed. βœ… Avoid unnecessary repeat attempts - You can request the export before the crawl finishes. To export the completed crawl, wait until the Issues page finishes processing and the results load. - Submit one export request at a time. - Record the first error and its time if the failure repeats. - Use the issues view to review available results while the export problem is investigated.

Last updated on Nov 10, 2026

πŸ” Reading Existing Site Audit Results Safely

πŸ”Ž What This Article Covers This guide explains how to read the status, page count, and URL issues of a Site Audit that has already run. Reading results does not start a new crawl or change the audit, so it is safe to use at any time. πŸ†” Finding Your Audit ID Every Site Audit has a numeric ID that appears at the end of its dashboard address. For example, in the address /site-audit/83952, the audit ID is 83952. To confirm an ID is valid for your account: 1. Sign in to the Search Atlas dashboard. 2. Open the address for the audit you want to check. 3. If the audit page loads with its crawl summary, the ID is accessible to the signed-in account, or through a valid public share link. That does not by itself prove the audit belongs to the account. 4. If you see a not-found response, the ID is not accessible to the signed-in customer account. Sign in to the account that owns the audit and try again. πŸ“Š Viewing Status and Page Count in the Dashboard The fastest way to read an audit's current state is in the dashboard itself: - Status: A finished audit shows a health result such as Healthy, Fair, or Poor. An active audit can show Crawling, Preprocessing, or Will recrawl soon, and a failed crawl shows Crawl failed. - Pages crawled: The audit list shows crawled pages against the crawl total, such as 630 of 1,500 pages. The overview shows the current crawled-page total. - URL issues: Issues are grouped by category and issue type, with Critical, Warning, and Notice severities. Open an issue to see the affected URLs. Viewing these pages does not affect the audit or use a new crawl. πŸ”Œ Reading Audit Data Through the API If you want to pull audit status, page counts, or URL issues into your own tools, you can use the Site Audit read endpoints. Follow these rules to avoid accidentally starting a new crawl: - Use a plain GET when you only want to read results. POST, PUT, and PATCH can create or change audits, and GET /api/v2/site-audit/{id}/issues/?export=true queues an issues export instead of only returning results. - Do not call crawl-triggering endpoints such as POST /api/v2/site-audit/{id}/recrawl/ when you only want existing results. A path containing start, create, or recrawl is not by itself proof that it starts a crawl. - Use your own API credentials. Generate or locate your API key or token in your account settings, and send it in the authorization header format shown in your API reference. - Use the Site Audit v2 paths in your API reference. Query parameters such as scope and segment_id change which results are returned, but the paths do not vary by plan. πŸ§ͺ Testing a Read-Only Request When you test your request, check that it is a GET request and that the audit ID in the path matches the audit you want. Replace the placeholders below with values from your own account and documentation: curl -X GET "https://[API_BASE_URL]/[SITE_AUDIT_READ_PATH]/83952" -H "Authorization: [YOUR_AUTH_HEADER]" -H "Accept: application/json" A successful detail response includes the audit status and crawl summary fields. The issues response instead returns issues grouped by category and issue type. If the response does not match the endpoint you called, compare the request with your API reference before trying again. ⚠️ Troubleshooting Common Errors - 401 or 403: Your credentials are missing, expired, or do not have permission to read this workspace. Generate a new key or check the workspace you are using. - 404: The audit ID does not exist in the workspace linked to your credentials. Confirm the ID in the dashboard first. - 405 or unexpected crawl activity: The request method or path may be wrong. Use GET only and remove any start or recrawl path. - Empty or partial results: The audit may still be in progress. Check the status in the dashboard and try again once it is complete. πŸ’¬ Need More Help? If you need further assistance, open the chat widget in the bottom-right corner of the platform and send your message. A member of our support team will reply in the chat.

Last updated on Nov 11, 2026

πŸ” False sitemap flags and score drops

⚠️ What this warning means A recent site audit can show inaccessible-sitemap issues, and the 0–1000 Technical Score (also called Site Health) can drop sharply, even when your pages look unchanged. The sitemap warning and the score are calculated separately. This article explains what each one means and what you can check before you contact support. βœ… Confirm your files are public Open your robots.txt and sitemap URLs in a browser, or use an external checker that is not Search Atlas. If those tools can read the files, the site itself is likely fine. In the case support reviewed, October 5 scans and outside checks still saw the sitemap and robots.txt while later Search Atlas scans warned that the sitemap was inaccessible. - Confirm robots.txt loads and lists your sitemap URL. - Confirm that sitemap URL opens and returns your URL list. - Compare the warning with an earlier audit that read the same files successfully. πŸ€– Why the audit can still flag the sitemap Support can recrawl the site and compare the crawler result with a normal browser request. Search Atlas flags a sitemap as inaccessible only after it requests that file and the request fails or the XML cannot be read. If robots.txt loads, the Sitemap URLs declared in it are queued to be fetched. A file that Search Atlas never requests is not marked inaccessible. If your browser can open the sitemap but the audit still flags it, the failure is in how Search Atlas fetched it, not something you fix by rewriting robots.txt or moving the sitemap. You cannot manually dismiss the inaccessible-sitemap flag. Leave publicly reachable files unchanged, then run a new audit. The new crawl requests the sitemap again, and a successful fetch clears the flag. If the new audit still reports it, contact support. πŸ“Š Why the health score can fall at the same time Support confirmed a scoring change around the week the score fell. Non-indexable pages were temporarily scored as zero and included in the site average, which lowered Technical Score even when the indexable pages had not changed. That score is tracked separately from the sitemap warning. After the scoring correction, the site average stopped counting those non-indexable pages, and a score can return above 900 when the indexable pages warrant it. Do not treat the score drop and the sitemap warning as one site error. The score change came from the updated scoring rules. The sitemap warning means Search Atlas requested that sitemap and the fetch failed or the file could not be read. πŸ› οΈ What you should do 1. Verify robots.txt and the sitemap with a browser or an external tool. 2. If both load, leave the files in place. Editing them will not fix a crawl that never requests the sitemap. 3. Note the date the warnings and the score change started, and whether an earlier audit still read the sitemap. 4. If the warning remains after a new audit, ask support to recrawl the site and review the sitemap fetch result, including the status recorded for that sitemap, against the file you can open in a browser. If you need further assistance, open the chat widget in the bottom-right corner of the platform and send your message. A member of our support team will reply in the chat.

Last updated on Nov 11, 2026

πŸ› οΈ Fixing a Stalled Site Audit Crawl

πŸ” Why Your Site Audit May Be Stalled If your project is not showing updated analyze stats while other projects update normally, the most common cause is that a firewall or security service on your website is blocking the Search Atlas crawler. When the crawler cannot access your site, the audit cannot complete and your stats stay out of date. Other common causes include: - Firewall or CDN blocking: Services like Cloudflare, Sucuri, or a host-level firewall may treat our crawler as a bot and block it. - robots.txt restrictions: Rules that disallow our user agent can prevent crawling. - Server issues: Downtime, slow response times, or rate limiting on your server can interrupt the crawl. - Large site size: Very large sites can take longer to crawl, which may look like a stall. 🧾 How to Identify the Search Atlas Crawler To let the crawler through your firewall, allow the user agent the project actually crawls with. By default that is SearchAtlas Bot (https://www.searchatlas.com). You can change it in Crawl Settings, so do not assume every crawl uses that agent, and do not rely on a fixed Search Atlas IP range β€” the crawler does not publish one and may use proxies. Open the project's Crawl Settings and check the User agent dropdown. The SearchAtlas option is noted as SearchAtlas Bot (https://searchatlas.com). Other choices are labeled by name only, and no IP ranges are shown. Copy the user agent you selected; there is no in-product IP list to copy. 🧱 Whitelisting the Crawler in Popular Firewalls Add the Search Atlas crawler to your allow list using the steps for your setup: - Cloudflare: Go to Security, then WAF, then Create rule. Set the rule to allow traffic where the user agent matches the Search Atlas crawler, then deploy the rule. - Sucuri or similar WAF: Add the crawler user agent or IP range to your whitelist in the firewall settings panel. - WordPress security plugins: Look for a bot whitelist or firewall exceptions setting and add the Search Atlas user agent there. - Host-level firewall: Ask your hosting provider to allow the Search Atlas crawler IP range for your domain. After updating your firewall, also check that your robots.txt file does not disallow the Search Atlas user agent, as this can block the audit even when the firewall allows access. πŸš€ How to Trigger a Fresh Crawl Once the crawler can access your site, start a new audit: 1. Open your project in Search Atlas. 2. Go to the Site Audit tool. 3. On the audit, click Recrawl Site to run a fresh crawl. If the site has never been crawled, the button says Start first crawl. Recrawl Page only recrawls that page. Use Crawl Settings only to change crawl options; changes apply on the next crawl. 4. Wait for the crawl to finish. Larger sites take longer, so allow time before checking results. 5. When the crawl finishes, open the project's Site Audit overview and review the updated results, including Site Health and the last crawled time. The figures update only after the crawl completes and can look unchanged if the site itself did not change. If the crawl still does not complete after whitelisting, run it once more to confirm the firewall change has taken effect, since some firewall rules can take several minutes to propagate. ⚠️ When to Contact Support If the crawl continues to stall after you have whitelisted the crawler and triggered a new audit, our team can investigate the crawl logs for your project. If you need further assistance, open the chat widget in the bottom-right corner of the platform and send your message. A member of our support team will reply in the chat. When contacting support, include your project name, the date of the stalled crawl, and a list of the firewall changes you have already made. This helps us resolve the issue faster.

Last updated on Nov 11, 2026

πŸ› οΈ Stop Unwanted Site Auditor Brand Loops

πŸ” What this issue looks like Some accounts see a website they do not want keep being audited in Site Audit, even though they did not start a new audit themselves. This can look like the account is being taken over, but the confirmed cases were a scheduled recrawl of an existing audit, not a change you made yourself. A different logo on the login page comes from the white-label domain you are signing in on, not from Site Audit. If a brand you do not recognize or do not want keeps returning in Site Auditor, do not try to force new projects or repeat audits on your own. The successful resolution was handled by a specialized team after the customer agreed to escalation. βš™οΈ How the issue was resolved In the confirmed resolution, support reviewed the account in Site Audit after the customer contacted them. There is no in-product control that assigns the account to a specialized monitoring team. Support can open Site Audit and check whether the unwanted website audit is still listed. There is no product feature that continues watching the account for the rest of the week. - The customer confirmed that the case could be escalated. - A specialized team monitored the account. - Support can check Site Audit and confirm whether the unwanted website audit is still listed. - Monitoring continued for the rest of the week. πŸ’¬ What you should do Contact support and clearly say that an unwanted brand is appearing in your account or Site Auditor and that you want it removed. Confirm that you agree to escalation so the specialized team can review it. If you need further assistance, open the chat widget in the bottom-right corner of the platform and send your message. A member of our support team will reply in the chat. βœ… After support confirms it is cleared Open Site Audit and confirm the unwanted website is no longer listed. If it returns, message support again in the same chat. Do not create the brand again in Brand Vault, and do not start a new audit for that website.

Last updated on Nov 12, 2026

πŸ” Fix Search Atlas Crawler Sitemap Access Issues

πŸ“Œ Overview Sometimes Site Audit reports that a sitemap is inaccessible, even though it worked before. Possible causes include a changed sitemap address, a security rule, a redirect, or a temporary fetch problem. This article explains how to check common causes. (content-assistant/app/utils/domain\_analysis\_utils.py:292–300, 358–365) ❓ What You May See - A message that no sitemap was found or the sitemap is not accessible. - Pages that were found in earlier crawls no longer appear in recent crawls. - The sitemap opens fine in your browser, but the crawler still reports a problem. 🧭 Why This Happens - The sitemap address is slightly different. Common names are sitemap.xml, sitemap_index.xml, and wp-sitemap.xml. A name like sitemaps.xml (with an extra s) is easy to mistype and may not exist on your site. - The sitemap moved or was renamed. An SEO plugin update, a theme change, or a site migration can change the sitemap address. - A security tool is blocking the crawler. Firewalls, CDNs, and bot protection can block automated visitors, even when the page loads normally for you. - The sitemap returns an error. The file might return a 404 (not found), 403 (forbidden), or 5xx (server error) message. - Too many redirects. A redirect from http to https, or from non-www to www, can cause problems if it loops or sends the crawler to the wrong place. - The file is not valid XML. Broken formatting, extra characters, or an error message inside the file can stop it from being read. - robots.txt is blocking access. Rules in your robots.txt file may block the sitemap or the crawler. πŸ› οΈ Step-by-Step Troubleshooting 1. Confirm the exact sitemap address. Open your site's robots.txt file (for example, yourdomain.com/robots.txt) and look for a line starting with Sitemap. This shows the official address of your sitemap. 2. Open the sitemap in a private browser window. If it shows a 404 or error page, the address is wrong or the file is missing. Update it to the correct address. 3. Check that the page loads as XML. You should see a list of URLs or a list of other sitemap files. If you see a normal web page, a login page, or an error message, fix that on your website first. 4. Check the final address after redirects. Make sure the sitemap loads in one step on your preferred version of the site (https and your chosen www or non-www version). 5. Review your robots.txt rules. Make sure there is no Disallow rule covering your sitemap file or blocking all crawlers from your site. 6. Check firewall, CDN, and security plugin settings. Look for bot protection, rate limiting, or country blocking that may stop the crawler. If you find blocked requests, allow the Search Atlas crawler and then run a new crawl. 7. Check your SEO plugin or CMS settings. Make sure sitemap generation is switched on and that the file is being created. Re-save your settings or regenerate the sitemap if needed. 8. Check the sitemap address on your website. If your robots.txt file references the sitemap, update that reference to the correct full URL, including https:// where applicable. Then run a new crawl. 9. Run a new crawl. After making changes, start a fresh crawl and check whether the sitemap is now found. βœ… Quick Checklist - The sitemap address is correct and matches what robots.txt lists. - The sitemap opens without a login and shows a successful result. - There are no redirect loops. - The file is valid XML and not empty. - robots.txt does not block the sitemap. - Your firewall or CDN is not blocking the crawler. - The correct address is saved in your Search Atlas project settings. πŸ’‘ Tips to Prevent This Issue - Use one standard sitemap address and avoid changing it often. - List your sitemap in robots.txt so crawlers can find it automatically. - Recheck your sitemap after plugin updates, redesigns, or migrations. - Review security and firewall settings after any change that affects bot traffic. - Keep your sitemap limited to live pages that return a normal status. πŸ’¬ Still Need Help? If the sitemap still cannot be accessed after these checks, please have the following ready: your project name, the full sitemap address, and the date of the most recent crawl that failed. This helps our team investigate faster. If you need further assistance, open the chat widget in the bottom-right corner of the platform and send your message. A member of our support team will reply in the chat.

Last updated on Nov 13, 2026

πŸ” Understanding Crawled Page Inventory

πŸ” What is crawled page inventory? Crawled page inventory is the collection of URLs that Search Atlas discovers and processes during a site crawl. It helps the platform understand your site structure, identify linking opportunities, and evaluate how pages connect to one another. This inventory is different from your website’s total number of pages, the number of pages indexed by search engines, or the number of URLs listed in your CMS. πŸ•·οΈ How are pages discovered? Search Atlas typically discovers pages using the crawlable links and site information available to the crawler, including: - Internal links found in page content, navigation, headers, footers, and other crawlable areas - XML sitemap URLs that are available to the crawler - URLs found while following links from previously discovered pages - Configured crawl settings, such as the starting URL, allowed pages, and crawl limits A URL may appear in a sitemap but remain unavailable for analysis if it cannot be fetched, is blocked, redirects unexpectedly, requires authentication, or falls outside the configured crawl scope. πŸ“Š What the inventory represents The inventory represents the pages Search Atlas could discover and successfully process during the relevant crawl. It may include pages with different statuses, such as successful pages, redirects, errors, or blocked URLs, depending on the report or feature being viewed. The inventory can change after a new crawl. Changes to internal links, sitemaps, robots.txt, redirects, canonical tags, site availability, or crawl settings may increase or reduce the number of available pages. πŸ”— How internal linking uses crawl data Internal linking features use the crawled inventory to identify relevant source and destination pages. Recommendations depend on the pages discovered, the content available for analysis, and the links already detected during the crawl. If a page is missing from the inventory, it generally cannot be used reliably as a source or destination for internal linking recommendations until it is discovered and processed successfully. πŸ› οΈ Why a page may be missing - The page is not linked from crawlable pages. - The URL is missing from the XML sitemap. - Robots.txt or page-level directives prevent crawling or indexing. - The page returns an error, redirect, timeout, or server response that prevents processing. - The page requires a login or other restricted access. - The URL is outside the project scope or crawl limits. - The page was added or changed after the last crawl. - URL variations, parameters, or canonical settings cause the crawler to treat it differently. βœ… How to improve crawl coverage 1. Confirm that the page returns a successful response when opened without authentication. 2. Add important pages to the XML sitemap. 3. Link to those pages from relevant, crawlable pages on the site. 4. Review robots.txt, noindex directives, canonical tags, and redirects. 5. Check the project’s crawl scope, limits, and starting URL. 6. Run a new crawl after making changes, then review the updated inventory. πŸ’‘ Important expectations Crawl inventory is a snapshot of what the crawler could discover and process at a specific time. It is not a real-time list of every page on the website and does not guarantee that a page is indexed by Google or another search engine. For the most accurate internal linking recommendations, keep your sitemap current, maintain clear internal navigation, and recrawl after significant site changes. πŸ’¬ Need help reviewing an escalation? When contacting support, include the project name, affected URL, crawl date, and any relevant crawl or sitemap details. This helps the team compare the page with the crawl results and determine why it was included or excluded. If you need further assistance, open the chat widget in the bottom-right corner of the platform and send your message. A member of our support team will reply in the chat.

Last updated on Nov 14, 2026

πŸ› οΈ Exclude WooCommerce Breadcrumb Schema

πŸ” Overview WooCommerce and MetaSync can both generate breadcrumb schema. On product pages, this may create duplicate or conflicting breadcrumb markup if both systems output the same schema. βš™οΈ Version 2.7.5 support In MetaSync version 2.7.5, there is no supported setting or built-in filter to exclude only WooCommerce product pages from MetaSync breadcrumb schema generation while keeping MetaSync breadcrumbs active on other pages. This means you cannot configure this product-page exception directly from the MetaSync settings in that version. 🧭 Available options - Use one breadcrumb source: If WooCommerce already provides the preferred breadcrumb schema on product pages, configure your site so only one system outputs breadcrumb schema there. - Review your theme and plugins: Check whether your theme or another SEO plugin adds WooCommerce breadcrumb markup. Multiple sources may cause duplicate schema. - Use custom development cautiously: A developer may be able to modify the generated markup or apply a site-specific integration. This is not a standard MetaSync setting or supported version 2.7.5 filter, so test changes carefully after updates. βœ… Recommended checks 1. Open a WooCommerce product page and inspect its page source or schema testing output. 2. Confirm whether WooCommerce, MetaSync, or another plugin generates the breadcrumb schema. 3. Check that the page contains only one valid breadcrumb schema graph. 4. Test non-product pages separately to confirm their MetaSync schema remains available. 5. Recheck representative product and non-product pages after any theme, plugin, or MetaSync update. πŸ“Œ Important considerations Do not remove breadcrumb navigation from the page solely to address duplicate schema. Visible navigation and structured data are separate. The goal is to keep the preferred breadcrumb markup while preventing duplicate or conflicting structured data. Because behavior can change between releases, verify the available controls when upgrading from version 2.7.5. A future release may introduce more specific WooCommerce exclusions or filtering options. πŸ’¬ Need help? If you need further assistance, open the chat widget in the bottom-right corner of the platform and send your message. A member of our support team will reply in the chat.

Last updated on Nov 14, 2026

πŸ› οΈ Troubleshoot OTTO Site Audit Crawl Failures

πŸ” Understanding the crawl failure ** When a Site Audit shows β€œCrawl has failed,” the audit did not complete successfully. The cause may be temporary or related to website access settings, security tools, server problems, or an internal processing issue. βœ… Check the crawl details 1. Open the affected project in Search Atlas. 2. Go to the OTTO site audit or crawl section. 3. Review the latest crawl status and any available error details. 4. Confirm that the crawl is running for the correct domain and protocol, such as HTTP or HTTPS. 🌐 Check website availability - Open the website in a browser and confirm it loads normally. - Check whether the site returns server errors such as 403, 429, or 5xx responses. - Confirm that the domain’s DNS records resolve correctly. - Make sure the hosting provider has not suspended the site or limited automated requests. πŸ”’ Review security and access settings Firewalls, CDNs, bot protection, and security plugins can block audit crawlers. Review these tools for blocked requests or rate limits. If your site uses an allowlist, add the crawler access required by your Search Atlas configuration. Also check your robots.txt file and any password, maintenance, or login requirement. The pages that need auditing must be publicly accessible to the crawler. βš™οΈ Retry the crawl 1. Fix any availability, security, or access issue you find. 2. Save the website changes and confirm the site loads successfully. 3. Start a new crawl from the OTTO audit area. 4. Allow the crawl time to finish before starting another retry. If the crawl fails again, compare the failure time with server, firewall, CDN, or hosting logs. Repeated failures at the same time can help identify a request block or server limit. πŸ“‹ Information to include when requesting help If the issue continues, note the project domain, the date and time of each failed crawl, the displayed status, and any error details. Also include recent changes to hosting, DNS, robots.txt, CDN, firewall, or security settings. If you need further assistance, open the chat widget in the bottom-right corner of the platform and send your message. A member of our support team will reply in the chat.

Last updated on Nov 15, 2026

πŸ” How to Run a New Website Audit

πŸ” Before you start After making website changes, open the existing audit and select **Recrawl Site** so Search Atlas crawls the site again and updates its SEO health data. - Confirm that your website is live and accessible. - Check that important pages are not blocked from crawling. - Make sure you are working in the correct Search Atlas project. πŸš€ Start a new audit 1. Log in to Search Atlas and open the project for your website. 2. Go to the Site Audit section. 3. Select **Audit Site** to create a new audit, or select **Recrawl Site** from an existing audit to crawl it again. 4. Review the website or crawl settings and update them if needed. 5. Start the audit and wait for the crawl to finish. The audit may take some time, depending on your website size and the number of pages being crawled. You can leave the page while the audit runs and return to review the results. πŸ“Š Review the updated results When crawling is complete, open the audit and review **Website Overview** for health statistics and crawl results, then open **Issues** to review issue categories. - Open individual issues for affected URLs and recommended fixes. - Compare the new results with the previous audit when available. - Prioritize critical technical issues before lower-impact recommendations. πŸ› οΈ If your changes are not visible If the report still shows an old issue, confirm that the updated page is live and that the audit crawled the correct URL. Some changes may also require additional time before search engines or third-party tools reflect them. Run another audit after confirming the changes are published and accessible to crawlers. πŸ’¬ Need help? If you need further assistance, open the chat widget in the bottom-right corner of the platform and send your message. A member of our support team will reply in the chat.

Last updated on Nov 16, 2026