🧭 Overview
A common misconception is that Search Atlas optimization fixes are applied client-side via JavaScript, meaning bots that do not execute JavaScript — such as GPTBot (ChatGPT), ClaudeBot (Anthropic), and PerplexityBot — would never see those changes. This is incorrect. Search Atlas uses a Cloudflare Worker with HTMLRewriter to transform your HTML before the response ever leaves Cloudflare's edge servers. Every crawler, including AI/LLM bots, receives fully modified HTML in the raw HTTP response — no JavaScript execution required.
⚙️ How the Cloudflare Worker Actually Works
When a request reaches your site, Cloudflare intercepts it at the edge. The Search Atlas Cloudflare Worker runs server-side using the HTMLRewriter API, which parses and rewrites the HTML response in a streaming fashion before it is delivered to the requester. This process happens entirely within Cloudflare's infrastructure — not in the visitor's browser.
This is fundamentally different from a client-side script tag that injects or modifies the DOM after the browser loads the page. HTMLRewriter operates on the raw HTML bytes at the network level. The requester — whether a human browser, Googlebot, GPTBot, ClaudeBot, or PerplexityBot — always receives the final, already-modified HTML.
🗂️ HTML Elements Modified by HTMLRewriter
The Search Atlas Cloudflare Worker uses HTMLRewriter to directly rewrite the following elements in your page's HTML before delivery:
<title>— Page title tags are updated to reflect optimized, targeted titles.<meta>tags — Meta descriptions, Open Graph tags, and other metadata are rewritten at the edge.<link rel="canonical">— Canonical link elements are corrected or injected to prevent duplicate content signals.<h1>,<h2>, and other heading tags — Heading content can be adjusted to align with on-page SEO recommendations.- img alt attributes — Missing or suboptimal alt text is added or updated directly in the HTML.
- Schema markup (JSON-LD) — Structured data blocks are inserted or modified in the
<head>or<body>of the document.
Because all of these changes happen at the edge before the response is sent, no JavaScript execution is needed by the recipient to see the optimized version of the page.
🤖 What AI Crawlers Actually Receive
AI and LLM crawlers such as GPTBot, ClaudeBot, and PerplexityBot send standard HTTP GET requests to fetch page content. They do not run JavaScript. However, because Search Atlas optimizations are applied before the HTTP response leaves Cloudflare, these crawlers receive the exact same fully optimized HTML that a browser would render.
To confirm this yourself, you can inspect the raw HTTP response of any page on your site using a tool like curl in your terminal:
- Run: curl -A "GPTBot" https://yoursite.com/your-page
- Look for your updated
<title>,<meta description>,<link rel="canonical">, and JSON-LD schema blocks in the raw HTML output. - These elements will reflect your Search Atlas optimizations without any JavaScript having been executed.
📍 Where to Monitor Your Optimizations in Search Atlas
You can review how your pages are being optimized and track AI crawler visibility directly inside the platform:
- In the left sidebar, navigate to AI Visibility to see how AI-driven crawlers are indexing and surfacing your content.
- Go to Site Metrics → Overview and select Activate AI Visibility if you have not already enabled this feature for your project.
- Use Site Metrics → Pages to inspect individual page-level metadata and confirm that your title, canonical, and schema changes are applied as expected.
❓ Frequently Asked Questions
- Does Search Atlas inject a script tag to make changes? No. The Cloudflare Worker uses HTMLRewriter to rewrite HTML server-side. There is no client-side script injection involved in applying metadata, canonical, heading, alt text, or schema changes.
- Do I need to update my CMS or codebase for this to work? No. The Cloudflare Worker operates independently at the network layer. Your CMS continues to serve pages as normal, and the Worker intercepts and modifies the response before it reaches any requester.
- Can I verify the changes are live? Yes. Use curl or any HTTP inspection tool to fetch your page's raw HTML and confirm the optimized elements are present in the response body.
- Are changes applied to every request, including cached responses? HTMLRewriter runs on responses passing through the Worker. Cloudflare's caching configuration for your site determines whether cached or fresh responses are used, but Worker transformations apply at the edge regardless.
💬 Need Help?
If you need further assistance, open the chat widget in the bottom-right corner of the platform and type human teammate to be connected with a member of our team.