The Crawl Monitoring feature provides real-time tracking and analysis of how search engines and AI bots interact with your website. This tool helps you improve site visibility and ensure efficient indexing by monitoring crawler behavior, distribution, and patterns.
You can access Crawl Monitoring directly from your Site Audit dashboard by selecting the Crawl Monitoring tab.
👣 Step-by-Step Instructions
👣 Step-by-Step Instructions
🖥️ Dashboard Components
🖥️ Dashboard Components
Crawler Distribution Table
Displays active crawlers (e.g., Google, Bing, Google-Mobile).
Shows total requests per crawler.
Includes interactive graphs for Historical Crawl Activity.
Historical Crawl Activity Graph
Presents crawler activity trends over time.
Color-coded by crawler type.
Toggle between Daily, Weekly, or Monthly views.
Hover over data points to see detailed metrics for specific dates.
Key Metrics
Site Indexation Percentage
Real-time representation of crawled vs. uncrawled pages.
Helps spot indexing gaps and opportunities.
Crawl Purpose Analysis
Discovery Crawls: When search engines index new pages for the first time.
Refresh Crawls: When search engines revisit existing pages for updates.
Provides distribution percentages to understand crawler intent.
Device Distribution
Breaks down activity by Desktop vs. Mobile crawlers.
Useful for ensuring mobile-first optimization.
Crawl Frequency Metrics
View patterns across:
Last 7 days
Last 30 days
Last 6 months
Last 1 year
Helps identify long-term trends in crawler behavior.
⚡ Best Practices
⚡ Best Practices
Monitor uncrawled pages: Keep an eye on the “Not Crawled” percentage to catch technical issues early.
Use device data for mobile SEO: Ensure mobile versions of pages are fully crawlable.
Leverage crawl frequency insights: Align your content update schedule with crawler activity trends.
Check crawler distribution balance: Confirm visibility across different search engines, not just Google.
❓ FAQs
❓ FAQs
❓ What’s the difference between discovery and refresh crawls?
❓ What’s the difference between discovery and refresh crawls?
💡 Discovery crawls happen when search engines find and index new pages for the first time, while refresh crawls revisit existing pages for updates.
❓ How often should I check Crawl Monitoring?
❓ How often should I check Crawl Monitoring?
⚡ Weekly reviews are recommended, but more frequent checks can help during active content updates.
❓ Can I see crawler activity broken down by device?
❓ Can I see crawler activity broken down by device?
✅ Yes, the Device Distribution metric shows how much crawler traffic comes from desktop vs. mobile bots.
Closing Note
✅ Crawl Monitoring gives you the visibility needed to understand how search engines and AI bots interact with your site. By tracking crawler behavior, distribution, and frequency, you can make smarter SEO decisions and ensure your site stays efficiently indexed.