🔍 Overview
Some customers notice that their site audit crawls complete the crawling phase but then show both crawl_state: failed and postprocessing_status: failed. In the same session, schema generation may consume quota credits for pages — sometimes 17 or more — while producing no visible output. This article explains what causes these issues, what they mean for your data, and what steps you can take right now.
⚙️ What Is Postprocessing and Why Does It Fail?
After Search Atlas finishes crawling your site, it runs a postprocessing stage that compiles raw crawl data into the audit issues, scores, and recommendations you see in your dashboard. If this stage fails, the audit cannot display results — even if pages were successfully crawled.
The most common reason postprocessing fails is a data conflict caused by concurrent activity. If a second crawl starts — or existing crawl data shifts — while the first crawl is still being processed, the system encounters rows it cannot reconcile and throws an internal error. This is a known platform issue and engineering has already deployed a fix for the most common crash scenarios (KeyError and IndexError during issue compilation).
Other factors that can contribute to postprocessing failures include:
- JavaScript-heavy pages: Sites with JS rendering can take significantly longer to crawl (median crawl time is over 13 hours in some cases). A crawl that appears stuck may still be running. Do not cancel and restart — this can trigger the concurrent conflict described above.
- Homepage detection errors: On certain site configurations, the system fails to identify the homepage correctly, resulting in zero pages crawled and a failed state. This is actively being resolved by engineering.
- Stale recrawl frequency settings: Some audits are stuck on a "never reprocess" state due to missing recrawl frequency data. A backfill fix is in progress to resolve this automatically.
📊 Why Does Schema Generation Consume Quota With No Output?
Schema generation quota is deducted at the point the generation job is triggered — before output is confirmed. If postprocessing fails or the crawl produces incomplete page data, the schema generation job may still fire for the pages that were queued, consume credits, and then fail silently because it has no valid page content to work with.
This means you can lose quota credits for pages — such as the 17-page scenario — without receiving any schema markup in return. The credits are consumed because the generation attempt was made, not because output was produced.
Your credits are not lost permanently. If schema generation fails due to a platform-side postprocessing error rather than invalid input on your end, the support team can review your account and arrange a credit adjustment where appropriate.
🚀 Steps to Take Right Now
- Check your crawl status: Go to Left sidebar → OTTO SEO → Site Audit → All Audits and review the status of the affected audit. If the status shows failed, do not immediately trigger a re-crawl.
- Wait before retrying: If your site uses JavaScript rendering, allow at least 14–16 hours before concluding a crawl is stuck. Cancelling early and restarting increases the risk of a concurrent conflict.
- Avoid running overlapping crawls: Do not start a new crawl for the same site while a previous crawl or postprocessing job is still active. Check Left sidebar → OTTO SEO → Site Audit → Crawl Monitoring to confirm all jobs for that site have fully completed before launching a new one.
- Retry the audit once: After confirming no other crawl is active for the site, go to Left sidebar → OTTO SEO → Site Audit → All Audits, locate the failed audit, and trigger a fresh crawl. The platform fix for postprocessing crashes is already deployed and should resolve the most common failure mode.
- Note your quota consumption: If schema generation consumed credits without producing output during the failed session, make a note of the affected site name, the date of the failed crawl, and the approximate number of pages affected. You will need this when contacting support for a credit review.
💡 What to Expect Going Forward
The engineering team has resolved the primary crash causing postprocessing failures. Additional fixes currently in development will address homepage detection errors and stale recrawl frequency settings. These will be applied automatically — no action is needed on your part.
If you re-run the audit and it fails again, or if you believe your schema quota was consumed unfairly, the support team can investigate your specific account logs and take action.
🆘 Still Need Help?
If you need further assistance, open the chat widget in the bottom-right corner of the platform and type human teammate to be connected with a member of our team.