🔍 Overview
When running a Site Audit on a competitor's domain, you may occasionally find that the audit cannot complete or returns an error. This is not a bug in Search Atlas — it is caused by the competitor's website actively blocking the audit crawler. This article explains why this happens and what steps you can take.
🚧 Why Some Competitor Domains Are Blocked
Many large websites use a content delivery network (CDN) called Amazon CloudFront to serve their content and protect their infrastructure. CloudFront includes security rules that can detect and block automated crawlers, including the Search Atlas Site Audit bot.
When a domain has these protections active, our crawler is denied access before it can gather any data. This is a deliberate security decision made by the website owner — it is entirely outside Search Atlas's control.
⚙️ What Happens During a Blocked Audit
If you attempt to audit a competitor domain that is protected by CloudFront or a similar security layer, the audit may fail to return meaningful data. This outcome does not indicate a problem with your Search Atlas account or project setup — it reflects the competitor's server actively rejecting crawler access.
✅ What You Can Do
While you cannot force access to a domain that actively blocks crawlers, there are several productive steps you can take:
- Confirm the block is on their end. Try accessing the competitor's sitemap directly in your browser (e.g., example.com/sitemap.xml). If it loads normally for you but the audit fails, the block is targeting automated tools specifically.
- Use alternative competitor research tools within Search Atlas. Other tools in the platform rely on Search Atlas's existing index rather than live crawling, meaning they are not affected by the competitor's crawler-blocking configuration. Explore the available research and analysis features to gather competitor intelligence without needing to crawl their domain directly.
- Monitor the domain over time. Security configurations change. You can retry the Site Audit on the competitor domain at a later date — some websites relax or update their crawler rules periodically.
- Focus on auditable competitors. Identify alternative competitors in your niche whose domains are not blocked, and use those audits to inform your strategy instead.
❓ Frequently Asked Questions
Is this a Search Atlas bug or limitation?
No. CloudFront blocking is a security measure applied by the website owner. Search Atlas has no ability to override or bypass another website's security infrastructure.
Can Search Atlas fix this for me?
Because the block is enforced by the competitor's server, there is no action our team can take to grant crawler access to a third-party domain. If you are unsure whether the issue is caused by blocking or something else, contact our support team with your project name and the domain you are trying to audit so we can investigate.
If you need further assistance, open the chat widget in the bottom-right corner of the platform and type human teammate to be connected with a member of our team.