Overview
When you add a client website to Site Explorer, Search Atlas crawls the domain to populate its report. If the site uses bot-protection measures — such as a CAPTCHA challenge — the crawler may be blocked mid-crawl. This can result in the project displaying an incorrect site name or image, and in some cases the project may appear to have failed even though your quota has been used.
This article explains what causes the issue, how to identify it, and what steps to take.
What Causes This Issue
Some websites use bot-detection services that trigger a CAPTCHA or access-denial page when automated crawlers visit them. When this happens during a Site Explorer crawl, the crawler captures the CAPTCHA page rather than the real website. As a result:
- The project may display an incorrect site title or thumbnail image.
- The report may appear incomplete or show unexpected data.
- Your project slot quota is consumed even though the crawl did not complete successfully.
This is a known platform behaviour. It cannot be resolved from within the platform UI, and there are no steps you can take on your side to fix the affected project. Engineering has already implemented improvements to detect and handle bot-protected pages more gracefully, so false site titles are less likely to be captured going forward.
How to Identify the Problem
You may be experiencing this issue if:
- You tried to add a client domain to Site Explorer and the project saved but shows incorrect or placeholder information.
- You see a site name or image that clearly belongs to a CAPTCHA or access-denied page rather than the real website.
- You attempted to add a second domain and received a quota-limit message, even though you believed you had available slots — this may indicate a previous crawl silently consumed a slot due to a bot-protection failure.
Steps to Take
- Confirm which domains are already saved. Open Site Explorer and review your existing projects. This helps you verify whether a domain was added (even if the crawl failed) and whether your quota has genuinely been consumed.
- Do not attempt to re-add the same domain repeatedly. Each attempt may consume an additional quota slot without producing a usable report.
- Check whether the client site uses bot protection. If you can confirm the site has a CAPTCHA or firewall service (such as Cloudflare's challenge page, hCaptcha, or similar), this is most likely the root cause.
- Contact support. Because this issue requires a manual fix on the back end — including restoring any quota incorrectly consumed — you will need to reach out to our team. If you need further assistance, please contact our support team directly so we can restore any affected quota and investigate the failed crawl.