🔍 What Is a Normal OTTO Crawl Time?

Most OTTO crawls complete within 15–45 minutes depending on site size and server response speed. Crawls on larger sites (5,000+ pages) or sites with slow hosting may take up to 60–75 minutes. A crawl running beyond 90 minutes is considered extended and requires escalation.

⚠️ User-Actionable Causes vs. Backend System Status

Before escalating, it helps to understand whether the delay is something you can influence or whether it requires backend investigation.

User-actionable causes (you can address these now):

• The site has a very large number of pages — OTTO crawls every accessible URL, so sites with 10,000+ pages naturally take longer.
• Slow server response times on your website — if your hosting is under load, OTTO receives pages more slowly.
• A previous crawl cycle was incomplete — starting a new crawl before the last one finishes can queue delays.
• Robots.txt or crawl-rate directives are throttling OTTO — check that your site does not restrict crawl frequency.

Backend system causes (require Support investigation):

• The crawl job is stuck in a processing queue behind other jobs.
• A backend worker has stalled mid-crawl.
• A system-side timeout has not yet triggered a retry.
• OTTO shows an active crawl status but no new pages are being processed.

Only a Support teammate with backend access can distinguish between these two categories with certainty.

🗺️ How to Check Your Current Crawl Status in the Platform

1. Go to OTTO SEO in the left sidebar, then open SEO Automation (URL: /seo-automation-v3).
2. Open the OTTO dashboard for the relevant website.
3. Look at the crawl progress indicator at the top of the page — it displays pages crawled, pages remaining, and an estimated completion time when available.
4. If the progress bar has not moved for more than 20 consecutive minutes, the crawl may be stalled rather than simply slow.
5. Note the exact time the crawl started (visible in the crawl history log below the progress bar) so you can report the duration accurately if you need to escalate.

Important: The platform progress indicator reflects the last-synced status. It may lag a few minutes behind actual backend processing. A crawl that appears frozen for under 20 minutes is likely still running normally.

📋 When to Provide a Time Estimate vs. Generic Guidance

Use the table below to decide the right response at each stage.

• Crawl running 0–60 minutes: No action needed. Share the normal range (15–45 min for small sites, up to 75 min for large sites) and advise the customer to wait.
• Crawl running 60–90 minutes with visible progress: Reassure the customer that large or complex sites can take this long. Confirm progress is still moving and give a rough estimate based on pages remaining.
• Crawl running 60–90 minutes with no visible progress for 20+ min: Escalate immediately — do not offer further generic troubleshooting.
• Crawl running 90+ minutes regardless of progress: Escalate immediately so a teammate can check backend queue position and provide an accurate time estimate or restart the job.

Only a Support teammate with backend access can provide a specific remaining-time estimate (for example, "35 minutes remaining"). If you cannot see this in the platform UI, do not guess — escalate instead.

🚨 Escalation Criteria and Steps

Escalate to the Support team when any one of the following is true:

• The crawl has been running for 90 minutes or longer.
• The progress bar has shown no movement for 20 or more consecutive minutes at any point during the crawl.
• The crawl status shows active but the pages-crawled counter has not increased in over 15 minutes.
• OTTO displays an error message or a spinning loader with no status text.

When you escalate, share the following information to speed up the investigation:

1. The domain being crawled.
2. The exact time the crawl started.
3. The current pages-crawled count and total pages shown in the UI.
4. Whether the progress bar has been moving recently or appears frozen.
5. Any error messages visible on the OTTO dashboard.

💡 Tips to Prevent Extended Crawl Times

• Avoid starting a new crawl while a previous one is still active — wait for full completion before re-triggering.
• Ensure your website's server is not under heavy load when running an OTTO crawl (avoid scheduling during peak traffic hours).
• Check that your robots.txt does not include a low Crawl-delay value that forces OTTO to slow down.
• For very large sites, expect the first crawl to take the longest — subsequent crawls are typically faster because OTTO only re-processes changed pages.

🙋 Need Help Right Now?

If your crawl has been running for 90+ minutes or appears stalled and you need an accurate status check or time estimate, please escalate directly to the team. If you need further assistance, open the chat widget in the bottom-right corner of the platform and type human teammate to be connected with a member of our team.