A Site Audit is a comprehensive analysis of your website's technical SEO health and performance. Using automated crawlers, it examines your website’s pages to identify issues that could affect rankings, user experience, and overall performance.
OTTO’s Site Audit tool is designed to spot areas of improvement for any live website. While no backend access is required to run an audit, connecting OTTO’s pixel is essential for deploying fixes based on audit results.
🐾 Step-by-Step Instructions
🐾 Step-by-Step Instructions
🔍 Setting Up Your Site Audit
🔍 Setting Up Your Site Audit
Navigation Path:
Left Sidebar → AI SEO → Site Audit
Here's a Loom with the exact steps!
Enter Your Website URL
Input the complete URL (e.g., https://www.example.com).
Verify that the URL includes the correct protocol (http:// or https://).
Configure Crawl Depth
Set the total number of pages you want analyzed.
Recommended: Total pages + 10% buffer.
Example: For 1,000 pages, set the limit to 1,100.
Set Crawl Frequency
Minimum: Monthly
For critical pages: Weekly
Consider factors like content update frequency, site size, and server capacity.
Choose User Agent
Simulate search engine behavior. Recommended:
Search Atlas User Agent
Googlebot Mobile
Adjust Crawl Speed
Faster = quick overview
Slower = more detailed insights
⚠️ Important: A slower crawl usually yields more accurate results.
Crawl Settings
Robots.txt Behavior:
Follow Robots.txt (Recommended): Matches Google’s crawl behavior.
Ignore Robots.txt: Crawls all accessible pages for full debugging.
JavaScript Rendering:
Available only on Pro & Enterprise plans.
Detects dynamic content and verifies OTTO fix implementations.
URL Exclusion Conditions (Beta)
Exclude irrelevant pages to optimize crawl budget.
Options:
Exclusion Rules: Block URLs containing specific terms (e.g., /blog).
Inclusion Exceptions: Add exceptions back (e.g., /blog/*).
Handle 404 pages by excluding them and re-running the audit.
⚙️ How to Exclude Pages from Crawling or Optimization in OTTO
⚙️ How to Exclude Pages from Crawling or Optimization in OTTO
1️⃣ Navigate to the OTTO SEO Area
In the left-hand navigation, click on AI SEO.
Select All Audits.
This takes you to the Site Auditor dashboard, where all audited sites are listed.
2️⃣ Open Site-Level Settings
Locate the site you want to configure in the Audited Sites list.
Click the three-dot (⋮) menu on the site card.
From the dropdown, select Crawl Settings.
💡 This menu is also where you can access GSC/GA Settings, trigger a Recrawl, or remove a site entirely.
3️⃣ Configure Crawl Settings
Inside the Crawl Settings modal:
Basic Controls
Max Pages to Crawl
Define the maximum number of URLs OTTO is allowed to crawl.Crawl Frequency
Choose how often OTTO should recrawl the site (e.g., Weekly).
Click Show more to access advanced options.
4️⃣ Set Advanced Crawl Rules (Critical Step)
In the expanded settings:
User Agent
Select the crawler identity (default: Google Chrome Desktop).Respect robots.txt
Toggle on/off depending on your crawl policy.JS Rendering
Enable if your site relies heavily on JavaScript.Crawl Speed
Control pages per second to avoid server strain.
5️⃣ Exclude Specific URLs or URL Patterns
Under URL Exclusion Conditions (Beta):
Click + Add condition.
Choose the rule type (e.g.):
Exclude → all URLs that contain
Enter the URL fragment or pattern you want excluded
Example:thebrothersbloom.com/florida
Add additional rules if needed (multiple exclusions are supported).
🚫 Any URL matching these conditions will be excluded from crawling and downstream OTTO optimizations.
6️⃣ Save and Apply
Click Update to apply changes.
OTTO will respect these exclusions on the next crawl cycle (or immediately if a recrawl is triggered).
✅ What This Means Operationally
Excluded URLs will not be analyzed
No on-page fixes will be generated or deployed
These pages are effectively removed from OTTO’s execution scope
This ensures precise control over which parts of your site OTTO can act on, reinforcing the platform’s principle of intentional, permission-based automation.
✅ Use cases for exclusions:
Sensitive content (private or restricted pages).
Pages under development.
Duplicate content that harms SEO.
⏱️ Post-Setup Audit Automation
⏱️ Post-Setup Audit Automation
Once the OTTO pixel is connected:
Go to Audit Scheduling in your OTTO dashboard.
Define frequency (daily, weekly, monthly).
Save your schedule.
OTTO will automatically monitor your site, ensuring consistent tracking of your SEO health.
❓ FAQs
❓ FAQs
Why do crawled pages not match total site pages?
Why do crawled pages not match total site pages?
💡 By default, audits crawl a limited number of pages. Adjust the page limit in settings to capture the full site.
Should I run the Site Audit before connecting the OTTO pixel?
Should I run the Site Audit before connecting the OTTO pixel?
⚡ Yes. Running the audit first ensures all site elements are detected correctly before integration.
Can I audit only certain sections of my site?
Can I audit only certain sections of my site?
✅ Yes. Use URL inclusion/exclusion rules to focus crawls on specific sections.
👉 Always run your Site Audit before creating OTTO projects. This ensures the crawler captures all pages correctly, avoids incomplete crawls, and sets up a smooth integration with the OTTO pixel later.