🔍 Overview
If Search Atlas is leaving your meta descriptions blank or injecting unexpected content — such as keywords or meta titles — into your pages, the most common cause is a crawler access restriction. When the Search Atlas crawler cannot reach your page, it cannot read or preserve your existing meta description, which causes it to fall back to a blank or auto-generated value.
This article walks you through diagnosing the problem and whitelisting the Search Atlas crawler IPs so your meta descriptions are processed correctly.
⚠️ Symptoms of This Issue
- Your meta description appears blank in search results or in the Search Atlas dashboard.
- Search Atlas is injecting keywords or the meta title into the description field against your instructions.
- The issue persists on specific pages even after you have manually set and saved a meta description.
- The problem affects one or more URLs but not your entire site.
🧠 Why This Happens
Search Atlas uses a crawler to read your pages and apply on-page optimisation settings, including meta descriptions. If your server, firewall, CDN, or security plugin blocks the crawler's IP addresses, the crawler cannot retrieve your existing meta description. As a result, Search Atlas either leaves the field blank or substitutes it with automatically generated content.
This is not a bug in your settings. It is a connectivity issue between the Search Atlas crawler and your web server that must be resolved at the server or security layer.
✅ Step 1 — Confirm the Crawler Is Being Blocked
Before whitelisting any IPs, confirm that a block is actually occurring:
- Log in to your hosting control panel, firewall dashboard, or CDN provider (for example, Cloudflare, Sucuri, or Wordfence).
- Navigate to the access logs or blocked requests section.
- Search for any blocked requests originating from the Search Atlas crawler.
- Note the timestamps and the specific URLs being blocked.
If you see blocked requests in the logs, proceed to Step 2. If logs are unclear or inaccessible, proceed directly to the whitelist step — adding the IPs causes no harm and resolves the issue in the majority of cases.
🔧 Step 2 — Whitelist the Search Atlas Crawler IPs
Add the Search Atlas crawler IP addresses to the allowlist in your firewall, CDN, security plugin, or hosting control panel. To obtain the exact list of current Search Atlas crawler IPs, please contact the support team using the instructions at the bottom of this article.
The exact steps for adding IPs to your allowlist depend on your platform. In general:
- Cloudflare: Locate the IP allowlist or firewall rules section in your Cloudflare dashboard and add each crawler IP as an allowed address.
- Wordfence: In your WordPress admin panel, navigate to the Wordfence settings and add each IP to the whitelist under the firewall allowlist options.
- Other firewalls or security plugins: Refer to your provider's documentation for adding IP addresses to an allowlist or trusted IPs list.
✅ Step 3 — Verify the Fix
- After whitelisting the IPs, return to the affected page in your Search Atlas dashboard.
- Trigger a re-crawl or wait for the next scheduled crawl cycle.
- Confirm that your meta description is now being read and preserved correctly.
If the meta description is still blank or incorrect after re-crawling, double-check that all crawler IPs were saved correctly in your firewall or security settings.
If you need further assistance, open the chat widget in the bottom-right corner of the platform and type human teammate to be connected with a member of our team.