Search Atlas Help Center

🕷️ Crawl Monitoring

Open in
View as Markdown Open in ChatGPT Open in Claude
Camilo Aponte

Camilo Aponte

Last updated on Sep 30, 2026

Use Crawl Monitoring to track which search engines and AI bots are crawling your site, how often they visit, and whether your pages are being discovered and indexed efficiently.

🖥️ Accessing Crawl Monitoring

Go to Left Sidebar → OTTO SEO → Site Audit → Crawl Monitoring.

📊 Dashboard Components

Crawler Distribution Table

  • Lists active crawlers (e.g., Googlebot, Bingbot, Google-Mobile) and their total request counts.
  • Includes interactive graphs for Historical Crawl Activity.

What to do next: If a major crawler like Googlebot is missing or underrepresented, confirm your site is indexable and submit your XML sitemap in Google Search Console.

Historical Crawl Activity Graph

  • Shows crawler activity trends over time, color-coded by crawler type.
  • Toggle between Daily, Weekly, or Monthly views.
  • Hover over data points to see detailed metrics for specific dates.

What to do next: If you see a sudden drop in crawl activity, check for recent robots.txt changes, server errors, or sitemap issues in the Site Audit tool.

Key Metrics

  • Site Indexation Percentage — Real-time representation of crawled vs. uncrawled pages. Use this to spot indexing gaps and opportunities.
    What to do next: If Site Indexation Percentage is below expectations, check your robots.txt and XML sitemap coverage in the Site Audit tool.

  • Crawl Purpose Analysis — Shows the split between:

    • Discovery Crawls: Search engines indexing new pages for the first time.
    • Refresh Crawls: Search engines revisiting existing pages for updates.

    What to do next: If Discovery Crawls are low after publishing new content, resubmit your sitemap and verify internal linking points to the new pages.

  • Device Distribution — Breaks down activity by Desktop vs. Mobile crawlers. Use this to verify mobile-first optimization coverage.
    What to do next: If Mobile crawler activity is low, run a mobile usability check in Site Audit to confirm responsive design and Core Web Vitals.

  • Crawl Frequency Metrics — View patterns across the last 7 days, 30 days, 6 months, or 1 year to identify long-term crawl trends.
    What to do next: If crawl frequency declines over time, audit page freshness, update key content, and check for crawl-budget waste from low-value URLs.

🤖 Crawl Analysis by Bot

The Crawl analysis by bot section gives you a visual breakdown of crawler activity per bot type, so you can see exactly which search engines and AI bots are spending time on your site.

  • Donut chart — Shows each bot's proportional share of total crawl requests. Hover over a segment to see the exact request count and percentage for that crawler.
  • Bot selector dropdown — Scroll through the dropdown to view every available crawler (e.g., Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot). Select a bot to filter the donut chart and associated metrics to that crawler only.
  • Legend — Lists each crawler by name with a matching color swatch. Use the color coding to map chart segments to specific bots.

To compare crawlers, switch between bots in the dropdown — the chart and legend update instantly.

What to do next: If an important crawler (such as Googlebot or a target AI bot) shows an unexpectedly low share, verify the bot is not blocked in robots.txt and review your Site Audit report for crawl errors. If AI bots are missing entirely, confirm your site allows their user agents.

📡 AI Bot Activity and Data Tracked

Crawl Monitoring tracks how search engines and AI bots interact with your website. Available data includes:

  • Total crawl requests
  • Average response time
  • Total download size
  • Crawler distribution
  • Historical crawl activity
  • Site indexation percentage
  • Crawl purpose
  • Device / screen size
  • Load time distribution
  • Crawl frequency
  • Crawled pages

❓ Why Crawl Monitoring Shows "No Data"

A No data message does not always indicate a problem. Data may not appear if:

  • The domain is new or has been live for less than approximately 6 months.
  • The domain has not yet had enough crawl activity.
  • The site has limited visibility or low crawler traffic.
  • Your account plan does not include Crawl Monitoring access (Pro or higher required).
  • Search engines or AI bots have not crawled the site during the selected date range.

✅ What to Check Before Reporting an Issue

  1. Confirm the selected Date Range covers a period with expected crawl activity.
  2. Switch between Daily, Weekly, and Monthly views to see if data appears at a different granularity.
  3. Check whether the domain is new or recently launched.
  4. Confirm the site is indexable and not blocked by robots.txt.
  5. Confirm your account plan includes Crawl Monitoring access (Pro or higher).
  6. Allow more time for crawler activity to accumulate — data builds as bots visit the site.

🎯 You now have a complete view of how search engines and AI bots interact with your site — from high-level crawler distribution to per-bot visual analysis in the Crawl analysis by bot section. To act on what you discover, explore the Site Audit tool in OTTO SEO to resolve crawl gaps and indexing issues.