📋 Overview
After completing a site migration—whether you changed domains, switched to HTTPS, restructured URLs, or moved platforms—your Search Atlas data may still reflect your old site. Running a fresh crawl ensures your audits, Site Explorer data, and on-page recommendations match your current website. This guide walks you through how to recrawl your entire site and what to do after the crawl finishes.
🕒 When You Should Recrawl
Recrawl your website whenever you have made significant changes, including:
- Migrating to a new domain or subdomain
- Switching from HTTP to HTTPS
- Changing your URL structure or permalinks
- Moving to a new CMS or hosting platform
- Publishing or removing large amounts of content
✅ Before You Start
To get accurate results, confirm the following first:
- Migration is complete. Wait until all redirects, DNS changes, and content updates are live.
- The site is publicly accessible. Remove any password protection, staging restrictions, or "noindex" directives that should not be there.
- Your robots.txt allows crawling. Make sure it does not block the pages you want analyzed.
- The correct domain is set in your project. If you migrated to a new domain, you may need to add it as a new project (see below).
🔁 Recrawling Within the Same Domain
If your site stayed on the same domain (for example, you only changed URL structures or content), you can recrawl your existing project:
- Log in to your Search Atlas Home.
- Open the Site Audit tool from the left-hand menu.
- Select the project for your migrated website.
- Click the Recrawl (or Crawl Now) button at the top of the audit page.
- Confirm the crawl settings, including crawl depth and page limits, then start the crawl.
The crawl runs in the background. Larger sites take longer, so allow time for it to finish before reviewing your results.
🌐 Recrawling After a Domain Change
If you migrated to a brand-new domain, create a new project so your data is tied to the correct URL:
- From your Search Atlas Home (Hi, CSM. What will we get done today?), click the Create Project button — or go to OTTO SEO → All Sites (OTTO SEO Projects) and click Create.
- Enter your new domain exactly as it appears live (including HTTPS and www, if applicable).
- Complete the project setup steps and connect any integrations, such as Google Search Console.
- Open Site Audit for the new project and start your first crawl.
Domain reprocessing for a migrated site now runs automatically once your project is set up—you no longer need to submit a manual request to our support team to have the domain recrawled.
⚙️ Adjusting Crawl Settings
For larger or recently migrated sites, review these settings before crawling:
- Page limit: Increase this if your site has more pages than the default allows, so the full site is captured.
- Crawl depth: Set a depth that reaches deeper pages in your site structure.
- Subdomains: Enable subdomain crawling if your content lives across multiple subdomains.
🏁 After the Crawl Finishes
Once the crawl completes, take these steps to confirm everything migrated correctly:
- Check the crawled page count. Compare it to your expected number of live pages.
- Review redirects. Look for broken redirect chains or 404 errors from old URLs.
- Scan for indexing issues. Confirm important pages are not blocked or marked "noindex."
- Look at site health scores. Use the audit recommendations to fix any new issues introduced during migration.
🛠️ Troubleshooting
🔍 The crawl returns very few pages
This usually means crawling is being blocked. Check your robots.txt file, remove any unintended "noindex" tags, and confirm the site is not behind a login or staging firewall.
🔗 Old URLs still appear in reports
Historical data is retained for reference. After a fresh crawl, new reports reflect your current site. If you changed domains, make sure you are viewing the new project.
🤖 The AI agent says the recrawl finished, but results haven't updated
If you triggered the recrawl through the in-app AI agent, confirm the crawl status directly in the Site Metrics tool rather than relying on the agent's confirmation. We are aware of an issue where the agent can report a recrawl as "completed" before it has actually verified the crawl's status. Treat the crawl status shown in Site Audit as the source of truth.
⏳ The crawl is taking a long time
Large sites naturally take longer. Crawls run in the background, so you can leave the page and return later. If a crawl appears stuck for an unusually long time, contact our support team.
💬 Need More Help?
If your results still look incorrect after recrawling, reach out to our support team through the in-app, typing "Human Teammate". Include your project name, your old and new domains, and the date of your migration so we can assist you quickly.