🗺️ Overview
When you delete URLs from OTTO, three things need to happen in the correct order: your sitemap and robots.txt must reflect the changes, Google Search Console must be notified, and OTTO must complete a new analysis cycle to recognise the updated state of your site. This article walks you through each step.
✅ Step 1: Confirm Your Sitemap and Robots.txt Are Updated
Before touching Google Search Console, verify that the files on your server already reflect the deletions.
- Sitemap: Open your sitemap URL (typically yourdomain.com/sitemap.xml) and confirm the deleted URLs no longer appear.
- Robots.txt: Open yourdomain.com/robots.txt and check that you have not accidentally blocked pages you still want indexed.
If OTTO manages your robots.txt, any rules you applied inside OTTO should already be saved. If you edited the file manually, make sure the file is live on your server before continuing.
🤖 Step 2: Configure the SearchAtlasBot User-Agent in Robots.txt
OTTO uses its own crawler called SearchAtlasBot to audit your site. For OTTO to crawl and analyse your pages accurately, this user-agent must not be blocked in your robots.txt file.
Check your robots.txt for any Disallow rules that apply to SearchAtlasBot or to all bots via a wildcard (User-agent: *). A correctly configured entry looks like this:
- User-agent: SearchAtlasBot
- Disallow: (leave blank to allow full access)
If SearchAtlasBot is blocked, OTTO will not be able to verify that your deleted pages are gone, and its analysis will not update correctly. Save and publish the robots.txt file once you have confirmed this entry.
📡 Step 3: Submit Your Updated Sitemap to Google Search Console
- Log in to Google Search Console and select the property that matches your domain.
- In the left sidebar, click Sitemaps under the Indexing section.
- If your sitemap is already listed, click on it and then click Resubmit. If it is not listed, enter your sitemap URL in the Add a new sitemap field and click Submit.
- Google will queue your sitemap for recrawling. The status will show as Success once it has been read.
🔍 Step 4: Request a URL Inspection and Recrawl in Google Search Console
Submitting the sitemap tells Google your site has changed, but for priority pages you want deindexed quickly, use the URL Inspection tool to speed up the process.
- In Google Search Console, paste a deleted URL into the URL Inspection search bar at the top and press Enter.
- Once the inspection loads, click Request Indexing.
- Google will add the URL to its crawl queue. This typically signals Google to check whether the page still exists and update its index accordingly.
Repeat this for any high-priority deleted URLs. Note that Google processes these requests within hours to a few days, depending on your site's crawl budget and authority.
⏱️ Step 5: Understand OTTO Recrawl Timing and the Next Analysis Cycle
After you update your sitemap, robots.txt, and Google Search Console, OTTO needs to run a new analysis cycle before it reflects the changes in your dashboard.
- OTTO triggers a new crawl and analysis based on a scheduled cycle. Most sites are re-analysed within 24 to 48 hours after changes are detected or manually triggered.
- If you want to prompt OTTO to re-analyse sooner, open your OTTO project, navigate to OTTO SEO → Site Audit → Overview (Website Overview), and use the Recrawl Site button.
- Until the new cycle completes, previously deleted URLs may still appear in OTTO's issue list. This is expected behaviour and will resolve once the crawl finishes.
🔎 Step 6: Verify Page Status Using Page Explorer
Once OTTO has completed its next analysis cycle, use the Page Explorer tool to confirm that your deleted URLs are no longer tracked as active pages.
- Inside your OTTO project, open the Page Explorer tab.
- Search for one of the deleted URLs using the search or filter bar.
- Check the page's status. A deleted page should show a 404 or removed status, or it may no longer appear in the list at all.
- If a deleted URL still shows as 200 OK, the page may still be live on your server. Double-check that the deletion was fully published and that any caching layers (CDN, server cache) have been cleared.
Page Explorer gives you a ground-truth view of what OTTO's crawler sees, making it the most reliable way to confirm that your deletions have been processed correctly end to end.
📋 Quick Reference: Full Workflow Checklist
- Confirm deleted URLs are removed from your live sitemap.
- Verify SearchAtlasBot is not blocked in robots.txt.
- Resubmit or submit your sitemap in Google Search Console.
- Use URL Inspection to request recrawl for priority deleted URLs.
- Wait 24–48 hours for OTTO's next analysis cycle to complete.
- Check Page Explorer to confirm deleted pages show a 404 or removed status.
💬 Need More Help?
If you need further assistance, open the chat widget in the bottom-right corner of the platform and type human teammate to be connected with a member of our team.