🔎 What the Warning Means
Site Audit reads your robots.txt file, a small text file that tells bots which parts of your site they may visit. If the file blocks AI and LLM crawlers (the bots used by tools like ChatGPT, Claude, Perplexity and Google's AI features), Site Audit shows a warning.
This is a warning, not an error. Your site still works and can still rank in regular search results. The warning only tells you that AI tools may not be able to read your content, so your pages are less likely to be cited in AI-generated answers.
🤖 Which Crawlers Are Usually Involved
These are the AI-related bots most often blocked in robots.txt files:
- GPTBot, OAI-SearchBot and ChatGPT-User (OpenAI)
- ClaudeBot, Claude-User, Claude-SearchBot and anthropic-ai (Anthropic)
- PerplexityBot and Perplexity-User (Perplexity)
- Google-Extended (Google's AI products)
- CCBot, Bytespider, Applebot-Extended, cohere-ai and Amazonbot (other AI and data-collection bots)
🧐 How to Spot the Problem in Your robots.txt
Open your robots.txt by typing your domain followed by /robots.txt in your browser, for example yourdomain.com/robots.txt. Look for a bot name on a line that starts with User-agent:, followed by a line that says Disallow: /. That combination means the bot is blocked from your whole site.
Also check for a general rule that applies to every bot. A line reading User-agent: * followed by Disallow: / blocks everything, including AI crawlers and search engines. If you see this, fix it first, because it can hurt your regular search visibility too.
🛠️ How to Fix It on Simvoly
Simvoly builds and manages your robots.txt, so the change is made in Simvoly, not in Search Atlas. The exact menu names can change as the builder updates, so use these steps as a guide:
- Log in to your Simvoly account and open the website you want to change.
- Go to the site-level Settings and look for the SEO area or a robots.txt option.
- Open the robots.txt content. If an editor is available, you can change it directly.
- Find each AI bot you want to allow. Delete its User-agent and Disallow: / lines, or change Disallow: / to Allow: /.
- Leave any rules for private areas (such as admin or checkout pages) in place.
- Save the change and publish your site so the new file goes live.
- Reload yourdomain.com/robots.txt in your browser and confirm the update appears.
If you can't find a robots.txt editor in your Simvoly plan, or the file keeps reverting after you save, the blocking rules may be added by a Simvoly default or a setting on your domain provider or CDN. In that case, check those settings, or reach out through our chat widget (details below) and share your robots.txt so we can point you in the right direction.
⚖️ Should You Allow AI Crawlers?
It depends on your goals. Allowing them is usually a good idea if you want your brand and content to appear in AI answers and AI search results. Blocking them may make sense if you don't want your content used to train AI models.
You can also take a middle path. Many sites allow AI search and citation bots (such as OAI-SearchBot, Claude-SearchBot and PerplexityBot) while blocking training bots (such as GPTBot or Google-Extended). If you choose to keep some blocks, the warning will stay in your audit. You can treat it as informational.
✅ Re-Run Your Site Audit
- After publishing the change, confirm the live robots.txt shows your updates.
- Open Site Audit in Search Atlas and start a new crawl.
- Check the results once the crawl finishes. The warning should be gone if all AI crawlers are now allowed.
If the warning still appears, the audit may have used an older copy of your file. Wait a short while, then run it again.
📇 Can You Update a Directory Listing After It's Published?
Yes. Business information on directories such as Data Axle can be corrected after it is published, for example your business name, address, phone number, website or hours. A few things are worth knowing:
- Updates go through the source. Each directory has its own process, so the change is usually submitted again to that directory rather than edited live on the page.
- Changes are not instant. Directories and the sites that copy their data can take days or weeks to show new details.
- Keep details consistent. Use the exact same name, address, phone number and website everywhere so search engines and AI tools see one clear version of your business.
- Check your website link. If your site address or settings changed, update the listing so it points to the right place.
To make a change, open the listing in your Search Atlas account, update the details and resubmit it if the option is available. Some directories limit which fields can be edited or how often. If you can't edit a listing, let us know which directory and which detail you want to change.
💬 Need More Help?
If you need further assistance, open the chat widget in the bottom-right corner of the platform and type human teammate to be instantly connected with a member of our team.