<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
xmlns:content="http://purl.org/rss/1.0/modules/content/"
xmlns:wfw="http://wellformedweb.org/CommentAPI/"
xmlns:dc="http://purl.org/dc/elements/1.1/"
xmlns:atom="http://www.w3.org/2005/Atom"
xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
>
<channel>
<title>You searched for robots - Prerender</title>
<atom:link href="https://prerender.io/search/robots/feed/rss2/" rel="self" type="application/rss+xml" />
<link>https://prerender.io</link>
<description>Prerender. JavaScript SEO, solved with Dynamic Rendering</description>
<lastBuildDate>Mon, 27 Apr 2026 16:19:07 +0000</lastBuildDate>
<language>en-US</language>
<sy:updatePeriod>
hourly </sy:updatePeriod>
<sy:updateFrequency>
1 </sy:updateFrequency>
<generator>https://wordpress.org/?v=6.9.4</generator>
<image>
<url>https://prerender.io/wp-content/uploads/favicon-150x150.png</url>
<title>You searched for robots - Prerender</title>
<link>https://prerender.io</link>
<width>32</width>
<height>32</height>
</image>
<item>
<title>How to Index Your Website on Google for the First Time</title>
<link>https://prerender.io/blog/how-to-index-your-website-on-google/</link>
<dc:creator><![CDATA[Prerender]]></dc:creator>
<pubDate>Mon, 27 Apr 2026 05:01:44 +0000</pubDate>
<category><![CDATA[Technical SEO]]></category>
<guid isPermaLink="false">https://prerender.io/?p=8102</guid>
<description><![CDATA[Learn how to index a website (for the first time), fix its crawl and rendering issues, and make it visible for AI search, too.]]></description>
<content:encoded><![CDATA[
<p>Your new website is live: the domain resolves, the pages load, and the product team signed off. Yet when you search for it, nothing appears because your website needs to be indexed by Google for the first time.</p>
<p>After the hard work of launching a site, most teams focus on content or design improvements, not on indexing. Some teams don’t even know how to index a website for the first time, so this process easily slips through the cracks. Not because it doesn’t matter, but because it tends to only come up when performance is poor, or content isn’t visible on Google.</p>
<p>In this blog, we will speak about the process of getting indexed on a brand-new site. This applies if you’ve migrated frameworks, rolled out a microsite, or replatformed your marketing tech stack.</p>
<h2 class="wp-block-heading">How Google Finds and Indexes a New Website</h2>
<p>Before your site can appear in search results, Google processes it in a sequence of five stages.</p>
<h3 class="wp-block-heading">1. Discovery</h3>
<p>Discovery happens when Google first becomes aware of your URL. This usually happens through:</p>
<ul class="wp-block-list">
<li>External links from other websites.</li>
<li>XML sitemaps submitted via Google Search Console.</li>
<li>Previously known URLs associated with your domain.</li>
<li>Internal links within your site.</li>
</ul>
<h3 class="wp-block-heading">2. Crawling</h3>
<p>Once discovered, web crawlers (like <a href="https://prerender.io/blog/understanding-web-crawlers-traditional-ai/" type="link" id="https://prerender.io/blog/understanding-web-crawlers-traditional-ai/">Googlebot and OpenAI’s crawlers</a>) request the page. Crawling retrieves the raw HTML and identifies links to other pages. If your site has no inbound links and no submitted sitemap, discovery and crawling may take longer. While crawling is a crucial part of the indexing process, it alone does not guarantee indexing.</p>
<p><strong>Crawl budget is another key factor here</strong>. Search engines allocate a finite number of pages they’ll crawl on your site within a given timeframe; once that budget is exhausted, <a href="http://prerender.io/blog/how-prerender-renders-javascript-websites" type="link" id="http://prerender.io/blog/how-prerender-renders-javascript-websites">Googlebot stops</a>, regardless of how many pages remain unvisited.</p>
<p>For JavaScript-heavy sites, this becomes a serious problem: rendering JS takes longer and requires more processing than static HTML, burning through your crawl budget faster and leaving pages undiscovered or deprioritized for indexing. Unrendered pages also <a href="https://prerender.io/framework/react/" type="link" id="https://prerender.io/framework/react/">hide internal links</a>, meaning crawlers can’t follow them. <a href="https://prerender.io/benefits/bigger-crawl-budget/" type="link" id="https://prerender.io/benefits/bigger-crawl-budget/">Managing your crawl budget</a> so it’s spent on valuable, fully renderable pages is one of the most impactful things you can do for indexation.</p>
<p><strong>Read: <a href="https://prerender.io/blog/5-ways-to-maximize-crawling-efficiency-for-faster-indexation/" type="link" id="https://prerender.io/blog/5-ways-to-maximize-crawling-efficiency-for-faster-indexation/">5 Ways to Maximize Crawling and Get Indexed Faster.</a></strong></p>
<h3 class="wp-block-heading">3. Rendering</h3>
<p>After crawling, Google determines what the page is actually about. If your website relies on JavaScript to load content dynamically, Google may need to render the page before it can fully evaluate it. While Google can render JavaScript, it can fail or be incomplete. Rendering allows Google to see the final version of the content, not just the initial HTML response. If key elements only appear after scripts execute, indexing depends on successful rendering. If rendering fails or is incomplete, Google may not see the page the way users do.</p>
<p><strong>Related: <a href="https://prerender.io/blog/why-javascript-complicates-indexation-and-how-to-fix-it/" type="link" id="https://prerender.io/blog/why-javascript-complicates-indexation-and-how-to-fix-it/">Why Does JavaScript Complicate My Indexing Performance?</a></strong></p>
<h3 class="wp-block-heading">4. Indexing</h3>
<p>After crawling and rendering, Google evaluates the page’s content, structure, metadata, and overall quality. If the page meets its indexing criteria, it is added to Google’s index: a massive, distributed database of searchable documents. Only once a page reaches the indexing state can it rank. Essentially, then it will appear on Google SERPs for a particular keyword or topic.</p>
<p>If a page is crawled but not indexed, it may appear in Search Console as “Discovered” or “<a href="https://prerender.io/blog/understanding-googles-crawled-currently-not-indexed-coverage-guide/" type="link" id="https://prerender.io/blog/understanding-googles-crawled-currently-not-indexed-coverage-guide/">Crawled – currently not indexed.</a>” In those cases, Google has seen the URL but has chosen not to store it.</p>
<p>In simple terms, every website moves through the same progression:</p>
<p><strong>Discovery → Crawling → Rendering → Indexing → Ranking.</strong></p>
<p>If your website isn’t showing on Google, something in that progression is breaking. Now let’s walk through the practical steps to make sure your site moves cleanly from discovery to indexing.</p>
<h2 class="wp-block-heading">Step-by-Step: How to Get Your Website Indexed on Google</h2>
<p>If you’re trying to make your website visible on Google for the first time, follow these steps in order.</p>
<h3 class="wp-block-heading">Step 1: Create and Verify Google Search Console</h3>
<p>Register your domain in Google Search Console.</p>
<ul class="wp-block-list">
<li>Add your domain as a property.</li>
<li>Verify ownership (via DNS, HTML file upload, or Tag Manager).</li>
<li>Confirm that your pages are accessible.</li>
</ul>
<figure class="wp-block-image size-full"><img fetchpriority="high" decoding="async" width="624" height="401" src="https://prerender.io/wp-content/uploads/How-to-create-and-verify-your-domain-in-Google-Search-Console.jpg" alt="How to create and verify your domain in Google Search Console" class="wp-image-8108" srcset="https://prerender.io/wp-content/uploads/How-to-create-and-verify-your-domain-in-Google-Search-Console.jpg 624w, https://prerender.io/wp-content/uploads/How-to-create-and-verify-your-domain-in-Google-Search-Console-300x193.jpg 300w" sizes="(max-width: 624px) 100vw, 624px" /></figure>
<p>Verification doesn’t automatically index your website, but it gives you visibility into crawl activity, indexing status, and potential technical issues. It also allows you to submit sitemaps and request indexing directly.</p>
<h3 class="wp-block-heading">Step 2: Submit Your Sitemap</h3>
<p>A sitemap is a structured file that lists the URLs you want crawled and indexed.</p>
<figure class="wp-block-image size-full is-resized"><img decoding="async" width="624" height="219" src="https://prerender.io/wp-content/uploads/Example-of-an-XML-sitemap.jpg" alt="Example of an XML sitemap" class="wp-image-8112" style="width:624px;height:auto" srcset="https://prerender.io/wp-content/uploads/Example-of-an-XML-sitemap.jpg 624w, https://prerender.io/wp-content/uploads/Example-of-an-XML-sitemap-300x105.jpg 300w" sizes="(max-width: 624px) 100vw, 624px" /></figure>
<p>It helps Google:</p>
<ul class="wp-block-list">
<li>Discover new pages more quickly.</li>
<li>Understand your site’s structure.</li>
<li>Identify priority URLs.</li>
</ul>
<p>To submit your sitemap:</p>
<ol class="wp-block-list">
<li>Locate it (commonly <code><span style="color: #188038;">yourdomain.com/sitemap.xml</span></code>).</li>
<li>In Search Console, navigate to <strong>Sitemaps.</strong></li>
<li>Paste the URL and click <strong>Submit.</strong></li>
</ol>
<figure class="wp-block-image size-full"><img decoding="async" width="624" height="353" src="https://prerender.io/wp-content/uploads/How-to-submit-an-XML-sitemap.png" alt="How to submit an XML sitemap" class="wp-image-8113" srcset="https://prerender.io/wp-content/uploads/How-to-submit-an-XML-sitemap.png 624w, https://prerender.io/wp-content/uploads/How-to-submit-an-XML-sitemap-300x170.png 300w" sizes="(max-width: 624px) 100vw, 624px" /></figure>
<p>Submitting a sitemap does not guarantee indexing. But it improves discovery, especially for new domains with limited inbound links.</p>
<h3 class="wp-block-heading">Step 3: Request Indexing for Key Pages</h3>
<p>If you’ve just launched and want priority pages indexed quickly (homepage, core product pages, documentation, etc.), use the <strong>URL Inspection</strong> tool in Search Console.</p>
<ol class="wp-block-list">
<li>Paste the page URL into the search bar.</li>
<li>Review the current indexing status.</li>
<li>Click <strong>Request Indexing.</strong></li>
</ol>
<figure class="wp-block-image size-full"><img loading="lazy" decoding="async" width="624" height="188" src="https://prerender.io/wp-content/uploads/URL-unknown-to-Google.png" alt="URL unknown to Google" class="wp-image-8114" srcset="https://prerender.io/wp-content/uploads/URL-unknown-to-Google.png 624w, https://prerender.io/wp-content/uploads/URL-unknown-to-Google-300x90.png 300w" sizes="(max-width: 624px) 100vw, 624px" /></figure>
<p>This signals to Google that the page should be re-crawled and evaluated. It’s not immediate, but it’s typically faster than waiting for discovery alone.</p>
<h3 class="wp-block-heading">Step 4: Check That Your Pages Are Crawlable</h3>
<p>If your website is not showing on Google, one of these common blockers may be preventing indexing:</p>
<h4 class="wp-block-heading">Noindex Tag</h4>
<p>If a page includes a <a href="https://prerender.io/blog/understanding-google-noindex-rendering/" type="link" id="https://prerender.io/blog/understanding-google-noindex-rendering/">noindex directive</a>, Google is explicitly instructed not to store it in the index. Remove the tag from live pages you want indexed.</p>
<h4 class="wp-block-heading">Robots.txt Disallow</h4>
<p>If your <a href="https://prerender.io/blog/robots-txt-and-seo/" type="link" id="https://prerender.io/blog/robots-txt-and-seo/">robots.txt file blocks access</a> to certain paths, Googlebot cannot crawl those pages. Update the rules to allow crawling where appropriate.</p>
<h4 class="wp-block-heading">Password Protection</h4>
<p>Search engines cannot access gated or login-protected content. Ensure public pages are accessible.</p>
<h4 class="wp-block-heading">Staging Domain Configuration</h4>
<p>If canonical tags or environment settings still point to a staging or development domain, Google may ignore your production URLs. Confirm that your live domain is self-referencing.</p>
<h4 class="wp-block-heading">Incorrect Canonical Tags</h4>
<p>If a page points to another URL as the canonical version, Google may choose not to index it. Ensure canonical tags reflect the intended primary URL.</p>
<p><strong>Related: <a href="https://prerender.io/blog/sure-fire-ways-to-get-urls-deindexed-by-google/" type="link" id="https://prerender.io/blog/sure-fire-ways-to-get-urls-deindexed-by-google/">12 Reasons Your URLs Get Deindexed by Google and Their Solutions</a></strong></p>
<p>If you’ve completed these steps and your site still isn’t appearing in search results, the next question is usually timing.</p>
<h2 class="wp-block-heading">How Long Does Google Take to Index a Site?</h2>
<p>Indexing timelines vary depending on your domain history, technical setup, and how easily Google can process your content.</p>
<p>As a general guideline:</p>
<ul class="wp-block-list">
<li><strong>Brand-new domains:</strong> anywhere from a few days to several weeks.</li>
<li><strong>Established, high-authority domains:</strong> often within hours to a few days.</li>
<li><strong>JavaScript-heavy or dynamically rendered sites:</strong> unpredictable.</li>
</ul>
<p>Even if your sitemap is submitted and your pages are crawlable, indexing can still fail at the rendering stage. If that processing step is delayed or incomplete, indexing will slow down.</p>
<h2 class="wp-block-heading">The Technical Layer: What Happens When Google Crawls JavaScript Content</h2>
<p>Up to this point, the process sounds straightforward: Google discovers your page, crawls it, and indexes it.</p>
<p>But for many websites, there’s an additional step.</p>
<p>When Googlebot requests a page, it first receives the raw HTML response from the server. On a static site, that HTML contains the full content (headings, text, links, and metadata), allowing Google to evaluate it immediately. On dynamic sites, however, the initial HTML may be minimal. Visible content is assembled in the browser after JavaScript runs.</p>
<p>That means:</p>
<ul class="wp-block-list">
<li>The primary page content may not exist in the initial HTML response.</li>
<li>Internal links may only appear after scripts execute.</li>
<li>Canonical tags and metadata may be injected dynamically.</li>
</ul>
<p>In those cases, Google must render the page by executing JavaScript before it can evaluate it properly. Rendering does not always happen immediately after crawling. If scripts are complex, blocked, or error-prone, Google may process the page late, partially, or not at all.</p>
<p>From your perspective, the website is live. From Google’s perspective, the page may appear incomplete.</p>
<h2 class="wp-block-heading">When JS Rendering Becomes the Indexing Bottleneck</h2>
<p>If you’ve verified Search Console, submitted your sitemap, removed crawl blockers, and requested indexing, but pages still don’t appear, the issue is often rendering.</p>
<p>For JavaScript-heavy sites, indexing reliability depends on how content is delivered to crawlers. There are three common approaches to solving rendering-related indexing issues:</p>
<h3 class="wp-block-heading">1. Client-Side Rendering (CSR)</h3>
<p>With client-side rendering, content is generated in the browser after JavaScript executes. This is simple from a development perspective, but search engines must render the page before evaluating it. Indexing may be delayed or inconsistent.</p>
<h3 class="wp-block-heading">2. Server-Side Rendering (SSR)</h3>
<p>The server generates fully rendered HTML before sending it to the browser. This ensures search engines receive complete content immediately. However, <a href="https://prerender.io/blog/what-is-srr-and-why-do-you-need-to-know/" type="link" id="https://prerender.io/blog/what-is-srr-and-why-do-you-need-to-know/">implementing SSR</a> often requires architectural changes and ongoing infrastructure management.</p>
<h3 class="wp-block-heading">3. Dynamic Rendering</h3>
<p><a href="https://prerender.io/blog/when-you-should-consider-dynamic-rendering/" type="link" id="https://prerender.io/blog/when-you-should-consider-dynamic-rendering/">Dynamic rendering</a> serves a pre-rendered HTML version of the page specifically to search engine crawlers, while users continue to receive the interactive JavaScript experience. This approach improves crawl reliability without requiring a full front-end rebuild.</p>
<p><strong>Read: <a href="https://prerender.io/blog/other-rendering-options-vs-prerendering/" type="link" id="https://prerender.io/blog/other-rendering-options-vs-prerendering/">Explaining the Difference Between Prerendering and Other Rendering Options</a></strong></p>
<p>For teams that rely on modern JavaScript frameworks but want predictable indexing, dynamic rendering with a tool like <a href="https://prerender.io" type="link" id="https://prerender.io">Prerender.io</a> provides a practical middle ground.</p>
<h2 class="wp-block-heading">A Practical Approach to Dynamic Rendering with Prerender.io</h2>
<p>Prerender.io implements dynamic rendering at scale. It detects search engine crawlers and serves them a fully rendered HTML snapshot of each page, ensuring Google can crawl and index complete content without relying on client-side execution.</p>
<p>This stabilizes indexing for:</p>
<ul class="wp-block-list">
<li>AI-generated or no-code front-end platforms (such as <a href="https://prerender.io/blog/how-to-make-lovable-websites-seo-friendly/" type="link" id="https://prerender.io/blog/how-to-make-lovable-websites-seo-friendly/">Lovable</a>).</li>
<li><a href="https://prerender.io/framework/react/" type="link" id="https://prerender.io/framework/react/">React</a>, <a href="https://prerender.io/framework/vue-js/" type="link" id="https://prerender.io/framework/vue-js/">Vue</a>, <a href="https://prerender.io/framework/angular/" type="link" id="https://prerender.io/framework/angular/">Angular</a>, and <a href="https://prerender.io/framework/others-js/" type="link" id="https://prerender.io/framework/others-js/">Next.js</a> applications.</li>
<li>Headless CMS architectures.</li>
<li><a href="https://prerender.io/blog/how-to-optimize-single-page-applications-spas-for-crawling-and-indexing/\" type="link" id="https://prerender.io/blog/how-to-optimize-single-page-applications-spas-for-crawling-and-indexing/\">Single-page applications</a>.</li>
<li>Sites undergoing framework migrations.</li>
</ul>
<p>By delivering complete HTML from the start, rendering becomes predictable and no longer depends on when or how JavaScript executes. For teams launching new properties or replatforming existing ones, this creates a reliable indexing foundation without requiring a full front-end rebuild.</p>
<p><strong>Watch the video below to learn how Prerender.io works or <a href="https://prerender.io/pricing/" type="link" id="https://prerender.io/pricing/">explore our pricing options</a>.</strong></p>
<figure class="wp-block-embed is-type-video is-provider-youtube wp-block-embed-youtube wp-embed-aspect-16-9 wp-has-aspect-ratio"><div class="wp-block-embed__wrapper">
<iframe title="How Does Prerender.io Work? A Quick Explainer" width="640" height="360" src="https://www.youtube.com/embed/OxNt36HhCP4?feature=oembed" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>
</div></figure>
<h2 class="wp-block-heading">FAQs About How to Index Your Website on Google</h2>
<h3 class="wp-block-heading">1. What is the Difference Between Crawling and Indexing?</h3>
<ul class="wp-block-list">
<li><strong>Crawling </strong>is when Googlebot discovers and retrieves your page.</li>
<li><strong>Indexing </strong>is when Google evaluates and stores that page in its searchable database.</li>
</ul>
<p>A page can be crawled but not indexed. If you’re comparing crawling vs. indexing to diagnose why your website isn’t showing in Google, indexing is usually where issues appear, especially if rendering fails or quality signals are weak.</p>
<h3 class="wp-block-heading">2. Why is My Website Not Showing on Google?</h3>
<p>If your site isn’t appearing in search results, the indexing process likely broke at one of these stages:</p>
<ul class="wp-block-list">
<li>The page hasn’t been discovered.</li>
<li>It was crawled but not indexed.</li>
<li>A noindex tag or robots.txt rule is blocking it.</li>
<li>Canonical tags point elsewhere.</li>
<li><a href="https://prerender.io/framework/" type="link" id="https://prerender.io/framework/">JavaScript rendering</a> failed.</li>
</ul>
<h3 class="wp-block-heading">3. How Much Does Dynamic Rendering Cost?</h3>
<p>Dynamic rendering costs vary depending on traffic volume and crawl frequency. Prerender.io uses a <strong>usage-based pricing model</strong> with several plan tiers designed to scale with your site’s needs:</p>
<ul class="wp-block-list">
<li>Plans start at <strong>$49 per month</strong> and include a set number of renders per month.</li>
<li>Higher tiers (e.g., Growth and Pro) include larger monthly render allowances and more advanced features.</li>
<li>Additional renders beyond your plan’s quota are billed at a set rate per 1,000 renders.</li>
</ul>
<p>A “render” refers to a fully rendered version of a page that Prerender.io stores and serves to crawlers. <a href="https://prerender.io/pricing/" type="link" id="https://prerender.io/pricing/">Review Prerender.io’s pricing</a> for more information.</p>
<h3 class="wp-block-heading">4. How Can I Get My Website Indexed on Google Faster?</h3>
<p>Steps to get your website indexed faster include:</p>
<ul class="wp-block-list">
<li>Verifying Google Search Console.</li>
<li>Submitting a sitemap.</li>
<li>Requesting indexing via URL Inspection.</li>
<li>Improving internal linking.</li>
<li>Ensuring full content exists in the initial HTML.</li>
</ul>
<p>If content only appears after JavaScript execution, indexing may be delayed until rendering completes.</p>
<h3 class="wp-block-heading">5. Does Indexing Affect AI Search and AEO Visibility?</h3>
<p>Yes, indexing affects AI search visibility as it is the foundation for both traditional search rankings and Answer Engine Optimization. AI systems <a href="https://prerender.io/blog/understanding-web-crawlers-traditional-ai/" type="link" id="https://prerender.io/blog/understanding-web-crawlers-traditional-ai/">rely on indexed content</a> as a source layer. If your page is not indexed, it cannot be surfaced in AI tools like ChatGPT, Claude, Gemini, Perplexity, or Google’s AIO.</p>
]]></content:encoded>
</item>
<item>
<title>How to Troubleshoot Common Lovable and Cloudflare Integration Issues</title>
<link>https://prerender.io/blog/troubleshooting-lovable-cloudflare-integration/</link>
<dc:creator><![CDATA[Prerender]]></dc:creator>
<pubDate>Thu, 19 Mar 2026 09:57:37 +0000</pubDate>
<category><![CDATA[Javascript SEO]]></category>
<category><![CDATA[javascript]]></category>
<guid isPermaLink="false">https://prerender.io/?p=7738</guid>
<description><![CDATA[Explore common Lovable and Cloudflare integration problems and learn how to fix them, ensuring your Lovable site's SEO performance is optimized for search.]]></description>
<content:encoded><![CDATA[
<p>A solid Cloudflare integration is the foundation for resolving Lovable SEO issues. But when the Lovable–Cloudflare integration layer isn’t configured correctly, problems surface quickly: your pages load for users but not for crawlers, or your metadata exists but doesn’t get picked up. Ultimately, your content visibility becomes incredibly inconsistent.</p>
<p>For some teams, connecting Lovable to Cloudflare is straightforward. But for others, it can be a debugging session involving redirect loops, blocked crawlers, caching conflicts, and subtle Cloudflare bot rendering issues that aren’t immediately obvious. </p>
<p>This guide breaks down the most common Lovable and Cloudflare integration failures, and shows you exactly how to fix each one.</p>
<h2 class="wp-block-heading">TL;DR of Common Lovable and Cloudflare Integration Challenges and How to Fix Them</h2>
<p>If your Lovable site isn’t indexing, search engine crawlers aren’t seeing content, or you’re getting Cloudflare errors like 1000 or 502, the issue is usually in the Cloudflare integration layer—not necessarily with Lovable itself.</p>
<p>In most cases, the Lovable and Cloudflare troubleshooting comes down to:</p>
<ul class="wp-block-list">
<li>Removing the custom domain from Lovable and letting Cloudflare fully control routing</li>
<li>Using the Lovable-specific Cloudflare Worker (not the standard version)</li>
<li>Ensuring DNS records are proxied (orange cloud) and not using A records</li>
<li>Setting SSL to Full (Strict)</li>
<li>Disabling Rocket Loader, Signed Exchanges, and reviewing Bot Fight Mode</li>
<li>Confirming bot requests return rendered HTML (look for <code><span style="color: #188038;">x-prerender-request-id</span></code>)</li>
<li>Separating bot and human cache behavior</li>
</ul>
<p>If you’re using Prerender.io for your Lovable app, most “not working” reports also trace back to misconfigured Workers, blocked bots, caching conflicts, or DNS/SSL mistakes in Cloudflare.</p>
<p>When the integration is correct, bot traffic is intercepted at the edge, routed properly, and receives fully rendered HTML, while human users continue to load the SPA normally.</p>
<h2 class="wp-block-heading">Why Lovable Apps Aren’t SEO-Friendly by Default</h2>
<p>Lovable is an AI-powered software builder that generates web applications from natural language prompts. These apps are typically client-side rendered, which means that when Googlebot or an AI crawler visits your Lovable website, it only sees an empty website while JavaScript fills in content and metadata after load. </p>
<p>Here’s what search engine crawlers receive when they make requests on a Lovable site:</p>
<pre class="wp-block-code dark"><code><!DOCTYPE html>
<html>
<head><title>My Site</title></head>
<body>
<div id="root"></div>
<script src="/assets/index-abc123.js"></script>
</body>
</html></code></pre>
<p>Every route returns this identical empty web page shell without the content, meta tags, or internal links. This is the core problem with client-side rendering (CSR), and it creates specific problems that directly affect content visibility and traffic.</p>
<ul class="wp-block-list">
<li><strong>Crawl delays.</strong> Google uses a two-phase process: it crawls raw HTML first, then queues JavaScript-heavy pages for a separate rendering pass. That queue adds days to weeks of delay, and execution is not guaranteed, <a href="https://prerender.io/benefits/faster-indexation/" type="link" id="https://prerender.io/benefits/faster-indexation/">causing pages to appear inconsistently in search results</a>.</li>
<li><strong>Invisible meta tags.</strong> Per-page titles, descriptions, canonical URLs, and JSON-LD structured data set via <code><span style="color: #188038;">react-helmet-async</span></code> are all JavaScript-rendered. Crawlers that skip JS execution never see them, which means your rich snippets and structured data produce no SEO value.</li>
<li><a href="https://prerender.io/benefits/social-media-sharing/" type="link" id="https://prerender.io/benefits/social-media-sharing/"><strong>Broken social link previews.</strong></a> Facebook, LinkedIn, Twitter/X, Slack, Discord, and WhatsApp all fetch pages without executing JavaScript. They receive the empty shell and generate blank or generic link previews. Broken social link previews directly reduce click-through rates on every shared link.</li>
<li><strong>No internal link discovery.</strong> React Router links and programmatic navigation calls are invisible in raw HTML. Search engines cannot discover linked pages even when those pages exist in a sitemap.</li>
<li><strong>Wasted crawl budget.</strong> Crawlers download heavy JavaScript bundles for every URL and encounter the same empty shell across all routes, directly harming crawlability and indexability across your entire site.</li>
</ul>
<h2 class="wp-block-heading">How to Check Whether Your Lovable Website Is Having SEO Issues</h2>
<p>Lovable’s own SEO and AEO guidance documents the practical limitations of client-side rendering (CSR), and notes that metadata does not automatically update across routes unless you implement route-specific metadata management.</p>
<p>To confirm whether your site has common Lovable SEO problems, run:</p>
<pre class="wp-block-code dark"><code>curl -i -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" \
-H "Accept: text/html" https://your-lovable-site.com</code></pre>
<p>If the response body contains only <code><mark style="background-color:#282a36" class="has-inline-color has-white-color"><code><div id="root"></div></code></mark></code> with no real content, search engine crawlers see nothing.</p>
<figure class="wp-block-table custom_table_styles_1"><table class="has-fixed-layout"><thead><tr><th class="has-text-align-left" data-align="left">Feature</th><th class="has-text-align-left" data-align="left">Visible to crawlers without Prerender?</th></tr></thead><tbody><tr><td class="has-text-align-left" data-align="left">robots.txt</td><td class="has-text-align-left" data-align="left">Yes</td></tr><tr><td class="has-text-align-left" data-align="left">sitemap.xml</td><td class="has-text-align-left" data-align="left">Yes</td></tr><tr><td class="has-text-align-left" data-align="left">Per-page meta tags</td><td class="has-text-align-left" data-align="left">No</td></tr><tr><td class="has-text-align-left" data-align="left">Open Graph / social previews</td><td class="has-text-align-left" data-align="left">No</td></tr><tr><td class="has-text-align-left" data-align="left">JSON-LD structured data</td><td class="has-text-align-left" data-align="left">No</td></tr><tr><td class="has-text-align-left" data-align="left">Page content and headings</td><td class="has-text-align-left" data-align="left">No</td></tr><tr><td class="has-text-align-left" data-align="left">Internal links</td><td class="has-text-align-left" data-align="left">No</td></tr><tr><td class="has-text-align-left" data-align="left">Canonical URLs</td><td class="has-text-align-left" data-align="left">No</td></tr></tbody></table></figure>
<p>To fix Lovable SEO issues, use <a href="https://prerender.io" type="link" id="https://prerender.io">Prerender.io</a> to pre-render your Lovable app and deliver fully rendered content to crawlers. This ensures 100% visibility of your website’s content. <a href="https://prerender.io/blog/how-to-make-lovable-websites-seo-friendly/" type="link" id="https://prerender.io/blog/how-to-make-lovable-websites-seo-friendly/">We’ve explained how Prerender.io makes your Lovable website SEO-friendly here.</a></p>
<p>That said, you may run into challenges when integrating Prerender.io with Cloudflare and Lovable. Let’s take a closer look at how the integration works—and the most common blockers you might encounter.</p>
<h2 class="wp-block-heading">How the Lovable, Cloudflare, and Prerender.io Integration Works</h2>
<p>A Cloudflare Worker sits between your visitors and your Lovable app, solving the Lovable SEO issues caused by JavaScript rendering. Every request passes through it, and it routes traffic in one of two directions based on the User-Agent header.</p>
<h3 class="wp-block-heading">How Cloudflare Worker Identifies Bots</h3>
<p>The Cloudflare Worker checks incoming <code><span style="color: #188038;">User-Agent</span></code> strings against a <code><span style="color: #188038;">BOT_AGENTS</span></code> list covering:</p>
<ul class="wp-block-list">
<li>Search engines: Googlebot, Bingbot, Yandexbot, Applebot</li>
<li>Social crawlers: facebookexternalhit, Twitterbot, LinkedInBot, Slackbot, Discordbot</li>
<li>SEO tools: Semrushbot, Ahrefsbot, Screaming Frog</li>
<li>AI bots: GPTBot, ClaudeBot, PerplexityBot, Anthropic-AI</li>
</ul>
<p>When a match is found, the Cloudflare Worker fetches <code><mark style="background-color:#282a36;overflow-wrap: break-word;" class="has-inline-color has-white-color">https://service.prerender.io/https://your-domain.com/path</mark></code> using your <code><mark style="background-color:#282a36" class="has-inline-color has-white-color">X-Prerender-Token</mark></code> and returns the rendered HTML. An <code><mark style="background-color:#282a36" class="has-inline-color has-white-color">X-Prerender</mark></code> header check prevents an infinite loop, since Prerender.io itself needs to fetch your page in order to render it.</p>
<figure class="wp-block-image size-full"><img loading="lazy" decoding="async" width="624" height="217" src="https://prerender.io/wp-content/uploads/How-Cloudflare-Worker-identifies-bots-on-Lovable-apps.png" alt="How Cloudflare Worker identifies bots on Lovable apps" class="wp-image-7755" srcset="https://prerender.io/wp-content/uploads/How-Cloudflare-Worker-identifies-bots-on-Lovable-apps.png 624w, https://prerender.io/wp-content/uploads/How-Cloudflare-Worker-identifies-bots-on-Lovable-apps-300x104.png 300w" sizes="(max-width: 624px) 100vw, 624px" /></figure>
<p>As the diagram shows, bot traffic routes through Prerender.io and returns rendered HTML, while human traffic passes directly to the Lovable app. Neither group is aware of the other’s path.</p>
<h3 class="wp-block-heading">Why Lovable Requires a Different Cloudflare Worker</h3>
<p>Prerender.io offers two Worker variants: a standard Cloudflare Worker and a Lovable-specific one. The bot-detection logic is identical in both, but the difference lies in how each handles human traffic.</p>
<p>The standard Worker calls <code><mark style="background-color:#282a36" class="has-inline-color has-white-color">fetch(request)</mark></code> to pass human visitors through to an existing origin server. Lovable-hosted sites do not have an accessible origin server in that sense. Instead, the Worker attaches via Custom Domains, which makes the Worker itself the origin. Human traffic must be explicitly proxied to <code><mark style="background-color:#282a36" class="has-inline-color has-white-color">yourapp.lovable.app</mark></code> inside the Worker code.</p>
<p>Prerender.io provides a separate gist for this (<code><mark style="background-color:#282a36; overflow-wrap: break-word;" class="has-inline-color has-white-color">github.com/Lasalot/4e5b294438d5404a5fb4aa4f63313dfb</mark></code>). It requires two changes: your Lovable upstream URL and your domain. Deploying the standard Worker gist on a Lovable setup silently breaks the site for human visitors. Bots appear to work fine, which makes the problem easy to miss.</p>
<p>Learn more about how to connect <a href="https://prerender.io" type="link" id="https://prerender.io">Prerender.io</a> to your Lovable websites through Cloudflare in these integration guides:</p>
<p><a href="https://docs.prerender.io/docs/how-to-integrate-prerender-with-lovable-hosted-websites" type="link" id="https://docs.prerender.io/docs/how-to-integrate-prerender-with-lovable-hosted-websites">Prerender.io Lovable Integration Guide</a></p>
<p><a href="https://docs.prerender.io/docs/cloudflare-integration-v2" type="link" id="https://docs.prerender.io/docs/cloudflare-integration-v2">Prerender.io Cloudflare Integration Guide (v2)</a></p>
<h2 class="wp-block-heading">Common Lovable and Cloudflare Integration Issues</h2>
<p>The failure points below cover the majority of Lovable and Cloudflare integration issues reported around this stack. If <strong><em>Cloudflare prerender not working</em></strong> describes your situation, one of these sections will identify exactly why. Most of these are Cloudflare bot rendering issues, cases where Cloudflare’s own features interfere with Prerender’s ability to intercept and serve bot traffic correctly.</p>
<h3 class="wp-block-heading">1. Redirect Loops</h3>
<p><strong>Cause:</strong></p>
<p>The custom domain is configured inside Lovable while also being handled by the Cloudflare Worker. Lovable forces its own redirects, the Worker adds another routing layer, and the request loops indefinitely. If a domain is marked primary inside Lovable, other domains redirect to it, which interacts poorly with proxy-based setups.</p>
<p><strong>Fix:</strong></p>
<p>Remove the custom domain from the Lovable project’s Domains settings. Only <code><mark style="background-color:#282a36" class="has-inline-color has-white-color">yourapp.lovable.app</mark></code> should remain configured in Lovable. The Cloudflare Worker Custom Domain handles the public-facing domain entirely.</p>
<p>This domain conflict is the single most impactful prerequisite in the entire setup. Skip it and nothing else works correctly.</p>
<p><strong>Verify:</strong></p>
<pre class="wp-block-code dark"><code>curl -I -L https://app.example.com/</code></pre>
<p>The chain should resolve cleanly to a <code><mark style="background-color:#282a36" class="has-inline-color has-white-color">200</mark></code>. If you see repeated <code><mark style="background-color:#282a36" class="has-inline-color has-white-color">301</mark></code> or <code><mark style="background-color:#282a36" class="has-inline-color has-white-color">302</mark></code> responses cycling without resolution, the domain is still connected inside Lovable.</p>
<h3 class="wp-block-heading">2. AI Crawlers Not Indexing or Citing Your Lovable Site</h3>
<p><strong>Cause:</strong></p>
<p>Cloudflare blocks AI training crawlers by default on newly added domains. Without explicitly enabling access, AI crawlers like GPTBot and PerplexityBot cannot reach your content regardless of your Prerender configuration.</p>
<p><strong>Fix:</strong></p>
<p>Go to Account Home, select your domain, navigate to Overview, and select “Control AI crawlers.” Review which crawlers are blocked and enable access for the ones you want to allow. Confirm the relevant AI bot User-Agents (GPTBot, ClaudeBot, PerplexityBot, Anthropic-AI) are also present in your Worker’s <code><mark style="background-color:#282a36" class="has-inline-color has-white-color">BOT_AGENTS</mark></code> list so they route through Prerender and receive rendered HTML.</p>
<p><strong>Verify:</strong></p>
<p>Go to Security > Events in your Cloudflare dashboard, then filter for AI crawler User-Agents such as GPTBot or ClaudeBot. If you see challenge or block events against those crawlers, they are being stopped before the Worker runs. No events means they are passing through as intended.</p>
<h3 class="wp-block-heading">3. Social Link Previews From Lovable Apps Are Generic or Wrong</h3>
<p><strong>Cause:</strong></p>
<p>Facebook, X, and LinkedIn preview crawlers do not execute JavaScript. They only see the initial HTML returned by the server. If <a href="https://prerender.io/blog/benefits-of-using-open-graph/" type="link" id="https://prerender.io/blog/benefits-of-using-open-graph/">Open Graph</a> and Twitter Card meta tags are not present in the initial HTML for each route, previews fall back to generic defaults.</p>
<p><strong>Fix:</strong></p>
<p>Ensure Open Graph and Twitter Card tags are present in the initial HTML for every route, not injected by JavaScript after load. Confirm the social crawler User-Agents are in your Worker bot list. The Prerender Lovable Worker template includes facebookexternalhit, twitterbot, and linkedinbot by default.</p>
<p><a href="https://prerender.io/blog/how-to-fix-link-previews/" type="link" id="https://prerender.io/blog/how-to-fix-link-previews/">Learn more about common social link preview problems and how to troubleshoot them.</a></p>
<p><strong>Verify:</strong></p>
<p>Run the URL through Facebook Sharing Debugger, X Card Validator, and LinkedIn Post Inspector. Each tool shows exactly what its crawler found when it fetched the page, so you can see directly whether your Open Graph tags are present and rendering correctly.</p>
<h3 class="wp-block-heading">4. Cloudflare Error 1014 and Error 1000</h3>
<p><strong>Cause:</strong></p>
<p>Lovable runs on Cloudflare’s infrastructure. When your domain is also on Cloudflare and uses a CNAME pointing to <code><mark style="background-color:#282a36" class="has-inline-color has-white-color">yourapp.lovable.app</mark></code>, cross-account CNAME resolution through Cloudflare’s proxy triggers these errors. Cloudflare defines Error 1014 as “CNAME Cross-User Banned” and Error 1000 as “DNS points to prohibited IP.”</p>
<p><strong>Fix:</strong></p>
<p>Use Workers Custom Domains rather than standard DNS route-based Workers. In your Cloudflare dashboard, go to Workers & Pages, open your Worker, navigate to Settings, and add your public domain under Custom Domains. Cloudflare provisions the DNS automatically. Do not manually create a CNAME or A record pointing to <code><span style="color: #188038;">yourapp.lovable.app</span></code> alongside this, as that reintroduces the conflict.</p>
<p>If your setup requires traditional DNS, use a CNAME record rather than an A record. An A record pointing to a Cloudflare IP is a documented trigger for Error 1000, and community members running this integration consistently identify A records as the source of DNS failures.</p>
<p>Regardless of DNS approach, audit your Worker for forwarded IP headers. Before fetching the Lovable upstream, delete <code><mark style="background-color:#282a36" class="has-inline-color has-white-color">CF-Connecting-IP</mark></code> and any duplicated <code><mark style="background-color:#282a36" class="has-inline-color has-white-color">X-Forwarded-For</mark></code> headers. Passing these through a Cloudflare-proxied request is a documented cause of Error 1000. The Prerender Lovable Worker template handles this by default. If you are using a custom Worker, add the deletions explicitly.</p>
<p><strong>Verify:</strong></p>
<p>Run a bot User-Agent curl request and look at the response. If you are still seeing 1000 or 1014 errors, a conflicting DNS record is still in place. If you see 525 or 526, the SSL mode needs attention. A clean <code><mark style="background-color:#282a36" class="has-inline-color has-white-color">200</mark></code> with no error codes means the DNS conflict is resolved.</p>
<h3 class="wp-block-heading">5. Cloudflare Worker Not Intercepting Bot Traffic</h3>
<p><strong>Cause:</strong></p>
<p>Route patterns are misconfigured, the Worker is deployed to the wrong Cloudflare zone, or the failure mode is set to fail-closed, returning a 500 instead of falling through to the origin. A missing or misconfigured <code><mark style="background-color:#282a36" class="has-inline-color has-white-color">PRERENDER_TOKEN</mark></code> secret will also silently break bot routing.</p>
<p><strong>Fix:</strong></p>
<p>Confirm route patterns cover all URL variations for your domain:</p>
<figure class="wp-block-table"><table class="has-fixed-layout"><thead><tr><th class="has-text-align-left" data-align="left">Website type</th><th class="has-text-align-left" data-align="left">Correct route pattern</th></tr></thead><tbody><tr><td class="has-text-align-left" data-align="left">example.com</td><td class="has-text-align-left" data-align="left"><code>example.com/*</code></td></tr><tr><td class="has-text-align-left" data-align="left">www.example.com</td><td class="has-text-align-left" data-align="left"><code>www.example.com/*</code></td></tr><tr><td class="has-text-align-left" data-align="left">Both www and non-www</td><td class="has-text-align-left" data-align="left"><code>*example.com/*</code></td></tr><tr><td class="has-text-align-left" data-align="left">Any subdomain</td><td class="has-text-align-left" data-align="left">*.<code>example.com/*</code></td></tr></tbody></table></figure>
<p>Set the Worker failure mode to “Fail open (proceed)” so errors degrade gracefully rather than taking the site down.</p>
<p>Confirm the Worker has <code><mark style="background-color:#282a36" class="has-inline-color has-white-color">PRERENDER_TOKEN</mark></code> set as a secret and that the bot path sends <code><mark style="background-color:#282a36" class="has-inline-color has-white-color">X-Prerender-Token</mark></code>. The token is mandatory and its absence produces no visible error, just a silent failure to render.</p>
<p>For Workers Custom Domains, Cloudflare will not attach the domain if the hostname already has an existing CNAME DNS record. Confirm the Custom Domain is attached, active, and no conflicting CNAME remains.</p>
<p>If newly added bot User-Agents are being ignored despite multiple redeploys, a known Cloudflare propagation issue may be the cause. Force propagation by renaming the Worker or adding a version comment before redeploying.</p>
<p><strong>Verify:</strong></p>
<p>Run a curl request with a bot User-Agent and confirm the <code><mark style="background-color:#282a36" class="has-inline-color has-white-color">x-prerender-request-id</mark></code> header is present in the response:</p>
<pre class="wp-block-code dark"><code>curl -sI -A "Googlebot" https://app.example.com/ | grep x-prerender</code></pre>
<p>If <code><mark style="background-color:#282a36" class="has-inline-color has-white-color">x-prerender-request-id</mark></code> comes back in the headers, the Worker is routing correctly. If nothing returns, bot traffic is not reaching Prerender.io. You can also replicate this in Chrome DevTools by switching the User-Agent to Googlebot under Network Conditions and checking the document response headers directly.</p>
<h3 class="wp-block-heading">6. Bot Requests Returning 401, 502, or Looping</h3>
<p><strong>Cause:</strong></p>
<p>A 401 response means Prerender rejected the request due to an authentication failure. A 502 with “Prerender loop detected” means the Worker is sending Prerender’s own rendering requests back through the proxy. Both are distinct failures with distinct causes.</p>
<p><strong>Fix:</strong></p>
<p>For 401 errors, confirm <code><mark style="background-color:#282a36" class="has-inline-color has-white-color">X-Prerender-Token</mark></code> is present in the bot request path and that the token value matches what Prerender expects. Prerender returns explicit hint text for missing or invalid tokens in the Edge response.</p>
<p>For 502 loop errors, confirm your Worker excludes requests that already carry Prerender identification headers. The Worker should also skip static assets entirely and only send <code><mark style="background-color:#282a36" class="has-inline-color has-white-color"><code><mark style="background-color:#282a36" class="has-inline-color has-white-color">GET</mark></code></mark></code> or <code><mark style="background-color:#282a36" class="has-inline-color has-white-color"><code><mark style="background-color:#282a36" class="has-inline-color has-white-color">HEAD</mark></code></mark></code> requests for HTML documents to Prerender. The Prerender Lovable Worker template excludes common file extensions by default. If you are using a custom Worker, add those exclusions explicitly.</p>
<p><strong>Verify:</strong></p>
<p>Fetch a bot request and read the response code and body together. A <code><mark style="background-color:#282a36" class="has-inline-color has-white-color">401</mark></code> with authentication language in the body is a token problem. A <code><mark style="background-color:#282a36" class="has-inline-color has-white-color">502</mark></code> that mentions a loop is a request filtering problem. The response body in both cases is descriptive enough to point you at the right fix.</p>
<h3 class="wp-block-heading">7. Cloudflare Cache Serving the Wrong Content</h3>
<p><strong>Cause:</strong></p>
<p>The stack has three caching layers: Prerender.io’s cache, Cloudflare’s CDN edge cache, and the browser cache. Conflicts between those layers produce two distinct problems. Cloudflare caches the empty SPA shell and serves it to bots before the Worker intercepts the request, so bots receive the shell instead of rendered HTML.</p>
<p>Separately, Cloudflare caches prerendered HTML and serves it to human visitors, breaking SPA functionality.</p>
<p><strong>Fix:</strong></p>
<ul class="wp-block-list">
<li>Run Prerender interception at the Worker level so it happens before CDN caching</li>
<li>Remove any Page Rules with <code><mark style="background-color:#282a36" class="has-inline-color has-white-color">Cache Level: Cache Everything</mark></code> on routes handled by the Worker</li>
<li>Use the Worker’s Cache API with separate cache keys for bot and human traffic to prevent cross-contamination</li>
<li>Set no-store or private cache headers on prerendered responses to prevent Cloudflare from auto-caching them</li>
<li>When purging, handle Cloudflare’s cache (via “Purge Everything” or URL-based purge) and Prerender.io’s cache separately via the dashboard or Recache API</li>
</ul>
<p><strong>Verify:</strong></p>
<p>Fetch the same URL with a bot User-Agent and then with a regular browser User-Agent. The bot response should contain full, rendered HTML with real content and meta tags. The human response should contain the SPA shell. If both return the same thing, cache separation is not working.</p>
<h3 class="wp-block-heading">8. Rocket Loader Breaking JavaScript Rendering</h3>
<p><strong>Cause:</strong></p>
<p>Cloudflare’s Rocket Loader rewrites <mark style="background-color:#282a36" class="has-inline-color has-white-color"><script type=”text/javascript”></mark> tags to <mark style="background-color:#282a36" class="has-inline-color has-white-color"><script type=”text/rocketscript”></mark> to defer JavaScript execution. This is one of the most common Cloudflare configuration mistakes that breaks JavaScript-rendered applications. It produces <mark style="background-color:#282a36" class="has-inline-color has-white-color">Uncaught SyntaxError</mark> errors in cached snapshots and causes bots to receive broken, partially rendered pages that Google may index incorrectly or skip entirely.</p>
<p><strong>Fix:</strong></p>
<p>Disable Rocket Loader at Speed > Optimization > Content Optimization.</p>
<p>Adding <mark style="background-color:#282a36" class="has-inline-color has-white-color">data-cfasync=”false”</mark> to specific script tags is not a viable workaround for Lovable apps. The Vite build output is not directly editable, so attribute-level overrides cannot be applied.</p>
<p><strong>Verify:</strong></p>
<p>Fetch a page with a bot User-Agent and search the raw HTML for <mark style="background-color:#282a36" class="has-inline-color has-white-color">text/rocketscript</mark>. If it appears on any script tag, Rocket Loader is still rewriting your scripts and needs to be disabled.</p>
<h3 class="wp-block-heading">9. Bot Fight Mode Blocking Legitimate Crawlers</h3>
<p><strong>Cause:</strong></p>
<p>Cloudflare’s Bot Fight Mode runs before WAF rules and before Workers, so it cannot be bypassed with WAF Skip rules or Page Rules. When active, it issues Managed Challenges to search engine bots, returning challenge pages instead of letting requests reach the Prerender Worker.</p>
<p>Separately, Cloudflare automatically blocks AI crawlers on newly added domains by default.</p>
<p><strong>Fix:</strong></p>
<ul class="wp-block-list">
<li>Disable Bot Fight Mode at Security > Bots if it is blocking legitimate crawlers</li>
<li>Switch to Super Bot Fight Mode, which can be bypassed via WAF custom rules with a Skip action</li>
<li>Enterprise customers can use <code><span style="color: #188038;">cf.bot_management.verified_bot</span></code> to exempt verified search bots explicitly</li>
<li>To allow AI crawlers, go to Account Home, select your domain, navigate to Overview, and select “Control AI crawlers”</li>
</ul>
<p><strong>Verify:</strong></p>
<p>Check Cloudflare’s Security Events log (Security > Events) for challenge events against Googlebot, GPTBot, or other crawler User-Agents. Any challenged or blocked entries confirm Bot Fight Mode is running before your Worker and intercepting legitimate crawler traffic.</p>
<h3 class="wp-block-heading">10. SSL Misconfigurations</h3>
<p><strong>Cause:</strong></p>
<p>The wrong SSL mode causes Workers to fail when making outbound <code><span style="color: #188038;">fetch()</span></code> requests to Prerender.io or the Lovable upstream. Flexible mode in particular has a known bug where Worker subrequests to external HTTPS hosts fail with Error 525.</p>
<p><strong>Fix:</strong></p>
<p>Set SSL to Full (Strict) at SSL/TLS > Overview. Full (Strict) requires a valid certificate on the origin. If the origin certificate is invalid or expired, Cloudflare returns Error 526.</p>
<figure class="wp-block-table"><table class="has-fixed-layout"><thead><tr><th class="has-text-align-left" data-align="left">SSL mode</th><th class="has-text-align-left" data-align="left">Effect</th></tr></thead><tbody><tr><td class="has-text-align-left" data-align="left">Flexible</td><td class="has-text-align-left" data-align="left">Worker fetch() to external HTTPS hosts fails with Error 525. Avoid.</td></tr><tr><td class="has-text-align-left" data-align="left">Full</td><td class="has-text-align-left" data-align="left">Works but does not validate origin certificates. Acceptable for development.</td></tr><tr><td class="has-text-align-left" data-align="left">Full (Strict)</td><td class="has-text-align-left" data-align="left">Recommended. Returns Error 526 if origin certificate is invalid or expired.</td></tr></tbody></table></figure>
<p>Worker subrequests to <mark style="background-color:#282a36" class="has-inline-color has-white-color">service.prerender.io</mark> always use Full (Strict) SSL regardless of your zone’s SSL setting. This works correctly since Prerender.io has valid certificates, but it is useful to understand when debugging unexpected SSL errors.</p>
<p>Check the Cloudflare SSL/TLS overview panel and confirm the mode is set to Full or Full (Strict). Then run a bot User-Agent curl request and confirm no <code><span style="color: #188038;">525</span></code> or <code><span style="color: #188038;">526</span></code> errors appear in the response.</p>
<h3 class="wp-block-heading">11. DNS Proxy Mode Set Incorrectly</h3>
<p><strong>Cause:</strong></p>
<p>With DNS-only mode (grey cloud) enabled, traffic bypasses Cloudflare entirely. The Worker never runs, and bot requests go directly to the Lovable origin without passing through Prerender.io.</p>
<p><strong>Fix:</strong></p>
<p>Enable proxy mode (orange cloud) on all DNS records for your domain. When using the Workers Custom Domains approach, Cloudflare creates the correct proxied DNS mapping automatically when you attach a Custom Domain to the Worker. If you are managing DNS records manually, set every relevant record to proxied.</p>
<p><strong>Verify:</strong></p>
<p>Open the DNS dashboard in Cloudflare and look at the proxy status column for each hostname the Worker should handle. Every record should show the orange cloud. A grey cloud means that the hostname’s traffic is bypassing Cloudflare entirely, and the Worker will never run for requests to it.</p>
<h2 class="wp-block-heading">How to Troubleshoot Lovable and Cloudflare Integration Problems</h2>
<p>With the common Lovable and Cloudflare integration failure understood, this is how the ideal integration should look.</p>
<h3 class="wp-block-heading">A. Domain Configuration</h3>
<p>The most reliable approach is to let Cloudflare own the domain entirely and keep Lovable out of the routing chain. In order, that means:</p>
<ol class="wp-block-list">
<li>Nameservers pointed to Cloudflare (full setup)</li>
<li>Custom domain removed from Lovable. Only <mark style="background-color:#282a36" class="has-inline-color has-white-color">yourapp.lovable.app</mark> remains as upstream</li>
<li>Worker Custom Domain attached to your public domain. Cloudflare auto-creates DNS and SSL</li>
</ol>
<p>If your setup requires traditional DNS instead of Workers Custom Domains, use CNAME records with proxy mode enabled. Never use A records, as they point to Cloudflare IPs and are a documented trigger for Error 1000 with Lovable’s infrastructure. The two records you need are:</p>
<ul class="wp-block-list">
<li>@ (apex): CNAME pointing to <mark style="background-color:#282a36" class="has-inline-color has-white-color">yourapp.lovable.app</mark>, proxied (orange cloud)</li>
<li>www: CNAME pointing to <mark style="background-color:#282a36" class="has-inline-color has-white-color">yourapp.lovable.app</mark>, proxied (orange cloud)</li>
</ul>
<h3 class="wp-block-heading">B. Cloudflare Settings</h3>
<p>Cloudflare ships with several performance and security features that are on by default and conflict with how Prerender.io works. Rocket Loader defers JavaScript execution, which breaks Prerender’s rendering process. Signed Exchanges let Google serve cached versions of your pages directly, bypassing your origin entirely. Bot Fight Mode runs before Workers and can block search crawlers before they ever reach Prerender.</p>
<p>None of these are obvious culprits, which is why they cause silent failures that are hard to trace. Before the integration behaves predictably, each of the following needs to be confirmed:</p>
<ul class="wp-block-list">
<li>DNS records proxied (orange cloud)</li>
<li>SSL set to Full or Full (Strict) with valid origin certificate</li>
<li>Automatic Signed Exchanges disabled: Speed > Optimization > Other</li>
<li>Rocket Loader disabled: Speed > Optimization > Content Optimization</li>
<li>Bot Fight Mode reviewed and confirmed not blocking search crawlers</li>
<li>AI crawler blocking reviewed: Account Home > domain > Control AI crawlers</li>
<li>No Page Rules with <mark style="background-color:#282a36" class="has-inline-color has-white-color">Cache Level: Cache Everything</mark> on Worker routes</li>
<li>Worker failure mode set to “Fail open (proceed)”</li>
</ul>
<h3 class="wp-block-heading">C. Worker Configuration</h3>
<p>The single most common Worker mistake is deploying the standard Cloudflare gist instead of the Lovable-specific one. The standard gist passes human traffic through using <mark style="background-color:#282a36" class="has-inline-color has-white-color">fetch(request) </mark>, which does not work when the Worker is the origin. Use the Lovable gist, update the two required values, and confirm the token is stored as a secret rather than a plain environment variable.</p>
<ul class="wp-block-list">
<li><mark style="background-color:#282a36; overflow-wrap: break-word;" class="has-inline-color has-white-color">PRERENDER_TOKEN</mark> set as a Secret in the Worker’s Settings > Variables and Secrets</li>
<li>Lovable-specific Worker code deployed (gist <mark style="background-color:#282a36" class="has-inline-color has-white-color">4e5b294438d5404a5fb4aa4f63313dfb</mark>), not the standard Cloudflare Worker</li>
<li>Two “CHANGE THIS” values updated: Lovable upstream URL and domain</li>
<li>Route patterns covering all required URL variations</li>
</ul>
<h3 class="wp-block-heading">D. Prerender.io Settings for Lovable Applications</h3>
<p>Prerender captures a snapshot of your page at the moment the headless browser considers it ready. By default, that happens as soon as the page loads, which means async content that has not resolved yet will be missing from the cached HTML bots receive. The <mark style="background-color:#282a36" class="has-inline-color has-white-color">prerenderReady</mark> flag lets you control exactly when that snapshot is taken.</p>
<p>Beyond that, cache expiration and sitemap submission determine how fresh and how discoverable your prerendered pages are. Make sure the following are all configured before treating the integration as complete:</p>
<ul class="wp-block-list">
<li>Cache expiration configured per content type: shorter for dynamic pages, longer for static</li>
<li>Sitemap submitted for proactive cache warming</li>
<li><mark style="background-color:#282a36" class="has-inline-color has-white-color">window.prerenderReady = false</mark> set early in the app lifecycle, switched to <mark style="background-color:#282a36" class="has-inline-color has-white-color">true</mark> only when all content and meta tags are fully in the DOM</li>
<li>Custom status codes set via <mark style="background-color:#282a36" class="has-inline-color has-white-color"><meta name=”prerender-status-code” content=”404″></mark> on error pages</li>
</ul>
<h2 class="wp-block-heading">Fix Your Lovable SEO and Cloudflare Integration for Good</h2>
<p>Lovable gets your app built and shipped fast. But without a JavaScript prerendering layer, every page it produces is invisible to search engines, social crawlers, and AI platforms. The traffic, the indexing, the social previews, the citations from AI search tools: none of it happens until bots can read your content. This affects discoverability across every channel, and ultimately, your revenue.</p>
<p>Prerender.io is the lowest-friction way to fix Lovable SEO challenges. You are not rewriting your Lovable app, adding SSR infrastructure, or changing how Lovable builds. A single Cloudflare Worker routes bot traffic through Prerender.io’s rendering service, and every crawler that visits your site gets fully rendered HTML. Your users never notice a change.</p>
<p>The integration fails in predictable ways when misconfigured. Remove the custom domain from Lovable, deploy the Lovable-specific Worker gist with the correct upstream URL, and disable Cloudflare’s conflicting performance features. Get those three things right and the rest of the configuration falls into place.</p>
<p>If you are not already using Prerender.io with your Lovable app, <a href="https://prerender.io/pricing" type="link" id="https://prerender.io/pricing">start a free trial</a> and follow the <a href="https://docs.prerender.io/docs/how-to-integrate-prerender-with-lovable-hosted-websites" type="link" id="https://docs.prerender.io/docs/how-to-integrate-prerender-with-lovable-hosted-websites">Lovable integration guide</a> to get your pages indexed and your previews working today.</p>
<p>Other Lovable SEO blogs and guides that may interest you:</p>
<ul class="wp-block-list">
<li><a href="https://prerender.io/blog/how-to-make-lovable-websites-seo-friendly/" type="link" id="https://prerender.io/blog/how-to-make-lovable-websites-seo-friendly/">How to Make Your Lovable Website SEO-Friendly</a></li>
<li><a href="https://prerender.io/prerender-vs-lovablehtml/" type="link" id="https://prerender.io/blog/how-to-make-lovable-websites-seo-friendly/">Prerender.io vs. LovableHTML Comparison</a></li>
<li><a href="https://docs.prerender.io/docs/cdn-issues" type="link" id="https://docs.prerender.io/docs/cdn-issues">CDN Issues and Cache Conflicts</a></li>
<li><a href="https://docs.prerender.io/docs/how-to-test-your-site-after-you-have-successfully-validated-your-prerender-integration" type="link" id="https://docs.prerender.io/docs/how-to-test-your-site-after-you-have-successfully-validated-your-prerender-integration">How to Verify Your Prerender Integration</a></li>
</ul>
]]></content:encoded>
</item>
<item>
<title>Industry Study: What 100M+ Pages Reveal About How AI Chooses Your Content</title>
<link>https://prerender.io/blog/industry-study-how-ai-retrieves-content/</link>
<dc:creator><![CDATA[Prerender]]></dc:creator>
<pubDate>Wed, 04 Mar 2026 10:08:59 +0000</pubDate>
<category><![CDATA[AI SEO]]></category>
<category><![CDATA[Data and Studies]]></category>
<category><![CDATA[Enterprise SEO]]></category>
<category><![CDATA[data]]></category>
<category><![CDATA[enterprise]]></category>
<category><![CDATA[studies]]></category>
<category><![CDATA[thought leadership]]></category>
<guid isPermaLink="false">https://prerender.io/?p=7545</guid>
<description><![CDATA[A study across 100M+ pages to surface AI crawler behavior, citation analysis, and what this means for your marketing.]]></description>
<content:encoded><![CDATA[
<p>The brands winning in AI search aren’t always the ones with the best content. <strong>They’re the ones AI systems can actually read and trust.</strong></p>
<p>That’s the central finding from a study conducted by Prerender.io and OtterlyAI analyzing 100M+ pages across eight global enterprise brands. This article breaks down what the data shows, helping you pinpoint what content AI systems prefer, seven key findings to include in your content strategy, and the three structural changes that are moving the needle on AI search visibility.</p>
<h2 class="wp-block-heading">The Dataset: Eight Global Brands and 100M+ Pages</h2>
<p>To conduct this study, the Prerender.io team analyzed 100M+ pages across eight anonymized brands in Prerender.io’s database, pulling the URLs most frequently requested by ChatGPT over Q4 2025. These are real machine-to-site retrieval requests, as opposed to pageviews or rankings, which means every signal in this dataset reflects exactly what AI systems are actively looking for.</p>
<p>The brands were selected to represent a range of industries, company sizes, and geographies. Each brand averages a minimum of 75M+ page renders per year, ensuring the dataset is large enough to surface reliable patterns. The brands chosen span ecommerce, SaaS, automotive, sports, fashion, and government to evaluate a variety of industries.</p>
<figure class="wp-block-table"><table class="has-fixed-layout"><tbody><tr><td><strong>Brand</strong></td><td><strong>Industry</strong></td><td><strong>Most-Requested Page Types</strong></td></tr><tr><td>Brand 1</td><td>Multinational automotive company based in Europe</td><td>Informational blog posts, homepage</td></tr><tr><td>Brand 2</td><td>Leading jewelry brand headquartered in the US (ecommerce)</td><td>Homepage, FAQs, comparison guides, product blogs</td></tr><tr><td>Brand 3</td><td>International sports organization</td><td>Streaming pages, live event pages</td></tr><tr><td>Brand 4</td><td>Global athletic leisure brand in 50+ countries (ecommerce)</td><td>New collection pages, homepage, FAQs, discount pages</td></tr><tr><td>Brand 5</td><td>American department store (ecommerce)</td><td>Top-selling product pages, homepage</td></tr><tr><td>Brand 6 </td><td>Global API/SaaS platform headquartered in the US</td><td>Technical documentation (exclusively)</td></tr><tr><td>Brand 7</td><td>Government / legal entity based in Europe</td><td>Documentation, homepage</td></tr><tr><td>Brand 8</td><td>Global fashion retailer headquartered in Europe (ecommerce)</td><td>Homepages (US + international), store locator, product pages</td></tr></tbody></table></figure>
<p>The OtterlyAI team then supplemented this retrieval analysis with citation data, examining how these brands actually appear in AI-generated answers across ChatGPT, Perplexity, and Google AI Overviews. Seven findings emerged. Here’s what they show.</p>
<h2 class="wp-block-heading">Key Finding #1: AI Systems Reward Easily Answerable Pages</h2>
<p>This is the clearest finding from the study across each brand. The most-retrieved URLs across all eight brands fall into a surprisingly narrow set of content types:</p>
<ul class="wp-block-list">
<li><strong>Evergreen editorial content and buying guides</strong></li>
<li><strong>FAQs and comparison pages</strong></li>
<li><strong>Technical documentation and API references</strong></li>
<li><strong>Category and collection pages</strong></li>
</ul>
<p>And these pages share a common characteristic. <strong>They answer a specific “what,” “how,” or “why” question without requiring the reader, or the AI crawler, to navigate anywhere else.</strong></p>
<p>This is a significant reframe for marketing teams accustomed to thinking about content in terms of the funnel or site hierarchy. AI visibility is not about where a page sits in your navigation.<strong> Instead, it’s about whether the page can reliably serve as an answer on its own. </strong>Whether it’s a blog article answering a specific question, a comparison page performing a this vs. that analysis, or a product category page, the pages that AI systems return to repeatedly are the ones that deliver complete and clearly structured answers. This also supports why AI systems seem to prefer knowledge base and help center content: they typically answer a single question. </p>
<p><strong>Key takeaway: </strong>write your content in a clear, question-answer format, focus on optimizing your knowledge base documentation, and ensure pages answer a specific “what,” “how,” or “why” question. </p>
<h2 class="wp-block-heading">Key Finding #2: Address Questions Directly, Especially for Ecommerce Brands</h2>
<p>This analysis also surfaced differences in how AI crawlers and humans navigate pages—especially for ecommerce. For example, when humans shop, they browse product detail pages in detail. But when AI systems research products on ecommerce sites, that’s not always the case.</p>
<p>Instead, AI systems consistently prefer:</p>
<ul class="wp-block-list">
<li><strong>Category and collection pages to answer <em>“what are the best options?”</em></strong></li>
<li><strong>FAQ and sizing guides to answer <em>“what should I get?”</em></strong></li>
<li><strong>Store locators and availability pages to answer <em>“where can I buy this?”</em></strong></li>
</ul>
<p>Individual product pages still matter, but far less than traditional SEO priorities suggest. The AI-driven shopping journey starts earlier and at a higher level of abstraction. Users receive AI-generated summaries before they ever visit a brand’s site, and those summaries are built from category-level and editorial content, not PDPs.</p>
<p>This means the pages most critical to AI visibility are often the ones least optimized for it. Category pages, FAQs, and buying guides tend to be JavaScript-heavy, dynamically loaded, and lower on the priority list for technical maintenance, precisely where AI systems are most likely to encounter incomplete or missing content.</p>
<p><strong>Key takeaway</strong>: prioritize your category pages, FAQs, and buying guides for both content completeness and technical rendering. These are the pages AI systems are currently using to build shopping summaries. Focus your efforts here, rather than on PDPs alone.</p>
<p><strong><em>Further reading: </em></strong><a href="https://prerender.io/blog/ai-indexing-benchmark-for-ecommerce/"><em>AI Indexing Benchmark Report for Ecommerce</em></a></p>
<h2 class="wp-block-heading">Key Finding #3: Third-Party Sites Matter For Citations </h2>
<p>This is the finding that should concern brand and growth leaders most. </p>
<p>In OtterlyAI’s analysis of one of the world’s top athletic brands, OtterlyAI looked at over 1,300 citations across 25 test queries. Here’s the breakdown of the share of citations per source:</p>
<ul class="wp-block-list">
<li><strong>Independent review sites: 8.6%</strong></li>
<li><strong>Reddit: 7.2%</strong></li>
<li><strong>YouTube: 4.6%</strong></li>
<li><strong>Brand’s own domain: 4.2%</strong></li>
</ul>
<figure class="wp-block-image size-large"><img loading="lazy" decoding="async" width="1024" height="576" src="https://prerender.io/wp-content/uploads/image-149-1024x576.png" alt="" class="wp-image-7620" srcset="https://prerender.io/wp-content/uploads/image-149-1024x576.png 1024w, https://prerender.io/wp-content/uploads/image-149-300x169.png 300w, https://prerender.io/wp-content/uploads/image-149-768x432.png 768w, https://prerender.io/wp-content/uploads/image-149-1536x864.png 1536w, https://prerender.io/wp-content/uploads/image-149.png 1920w" sizes="(max-width: 1024px) 100vw, 1024px" /></figure>
<p>Evidently, the brand’s own domain holds less weight than community platforms like G2, Reddit, or YouTube. And this isn’t an isolated case. Broader research from OtterlyAI across 1M+ citations across ChatGPT, Perplexity, and Google AI Overviews confirms the same pattern across industries: community platforms and third-party sources capture the majority of AI citations, regardless of how strong the brand’s owned content is.</p>
<p>AI systems are designed this way. They increasingly want to sound confident in their answers, and often <a href="https://prerender.io/blog/podcast-how-toggl-is-adapting-to-the-ai-era/">prioritize third-party validation and community consensus over brand-owned marketing content</a>. That’s a newer structural difference with LLMs, and your strategy has to account for this.</p>
<p><strong>Key takeaway:</strong> Build a deliberate third-party presence strategy. Identify the review sites, forums, and media outlets AI systems are already citing in your category, and prioritize earning coverage there.</p>
<h2 class="wp-block-heading">Key Finding #4: Technical Accessibility Is Crucial for AI Visibility</h2>
<p>Across this dataset, one issue emerged as the most consistent barrier to AI visibility. Brands were inadvertently blocking AI crawlers with configurations originally written for traditional search engines.</p>
<p><strong>When AI crawlers can’t access your site, you’re not partially visible to AI systems—you’re completely invisible. </strong>This means no citations, no mentions, and no influence on AI-generated answers. This issue is more common now because AI crawlers are newer and less familiar to infrastructure teams than Googlebot, and they’re frequently caught by security policies that predate their existence.</p>
<p>Before any content or strategy work, confirm that your site is actually accessible to major AI crawlers. It’s the single highest-leverage action available to brands that have invested in content and are seeing no AI visibility, and it requires a conversation with your technical team, not a content brief.</p>
<p>For JavaScript-heavy sites specifically, the gap between what a human sees and what an <a href="https://prerender.io/blog/understanding-web-crawlers-traditional-ai">AI crawler can read</a> is often significant. Server-side rendering or prerendering (serving AI crawlers a fully rendered, stable version of each page) is often the most reliable fix.</p>
<p><strong>Key takeaway: </strong>have your technical SEO team evaluate your technical performance, particularly your JavaScript rendering and robots.txt.</p>
<p><em><strong>Further reading:</strong> <a href="https://prerender.io/blog/seo-audit-vs-geo-audit-for-site-health/">How to Conduct a GEO Audit on your Site</a></em></p>
<h2 class="wp-block-heading">Key Finding #5: Homepages Are Gateways, Not Destinations</h2>
<p>Homepages appear frequently among the top-requested URLs, and sometimes as the single most-retrieved page. But the retrieval pattern tells a specific story.</p>
<p>AI systems hit the homepage and then fan out immediately. The homepage appears to serve a defined function to confirm brand identity, establish authority, and act as a starting point for deeper retrieval. However, the sharp drop-off in retrieval volume after the homepage means that what happens next determines the majority of your AI visibility.<strong> A strong homepage paired with inaccessible supporting content is nearly as limiting as no homepage presence at all.</strong></p>
<p><strong>Key takeaway: </strong>treat your homepage as a crawl entry point, not a destination. The real AI visibility work happens on the supporting pages it leads to.</p>
<h2 class="wp-block-heading">Key Finding #6: You Need a Different AI Search Strategy for Each AI Platform</h2>
<p>Not all AI platforms behave the same way, and this has direct implications for where marketing teams focus their energy. </p>
<p>Proprietary data from OtterlyAI shows the following:</p>
<figure class="wp-block-table"><table class="has-fixed-layout"><tbody><tr><td><strong>AI Platform</strong></td><td><strong>Citation Rate</strong></td></tr><tr><td>Perplexity</td><td>97% of responses include a citation</td></tr><tr><td>Google AI Overviews</td><td>34% of responses include a citation </td></tr><tr><td>ChatGPT</td><td>16% of responses include a citation </td></tr></tbody></table></figure>
<p>Perplexity has a near-universal citation rate, meaning that most answers include a citation, and that you have a higher chance of being cited in their response. This implies that, for a Perplexity-focused strategy, if you create comprehensive, well-structured content across all detail pages, it has a real chance of appearing. </p>
<p>However, this is not equally the same for other AI platforms: Google’s AI Overviews has a 34% citation rate, whereas ChatGPT only has 16%. This means that you may need to adapt your AI search strategy accordingly. For AI Overviews, focus on cornerstone content and domain authority. While for ChatGPT, prioritize page speed—it doesn’t like waiting around for slow-loading pages. </p>
<p><strong>A single “AI search strategy” will underperform, and it may need to change every few months. </strong>A brand might have strong Perplexity visibility today and near-zero ChatGPT presence tomorrow—not because of content differences, but because each platform has different technical and editorial preferences.</p>
<p><strong>Key takeaway: monitor AI visibility by platform, not in aggregate.</strong> A blended view of “AI traffic” will mask where you’re winning and where you’re invisible. You can use AI search analytics tools to do this.</p>
<figure class="wp-block-image size-large"><img loading="lazy" decoding="async" width="1024" height="576" src="https://prerender.io/wp-content/uploads/image-152-1024x576.png" alt="" class="wp-image-7623" srcset="https://prerender.io/wp-content/uploads/image-152-1024x576.png 1024w, https://prerender.io/wp-content/uploads/image-152-300x169.png 300w, https://prerender.io/wp-content/uploads/image-152-768x432.png 768w, https://prerender.io/wp-content/uploads/image-152-1536x864.png 1536w, https://prerender.io/wp-content/uploads/image-152.png 1920w" sizes="(max-width: 1024px) 100vw, 1024px" /></figure>
<h2 class="wp-block-heading">Key Finding #7: International Brands Can Experience Greater Visibility Issues</h2>
<p>For brands operating across markets, AI systems don’t default to your primary locale. They actively retrieve language-specific URLs, country variants, and localized pages, and appear to match content language directly to the user’s query language.</p>
<p>A French user asking about your product in French will receive an answer drawn from your French-language content, not an English page.<strong> If your localized pages are incomplete, slow to load, or invisible to AI crawlers, that market segment is effectively unserved by AI search</strong>—regardless of how strong your primary-market presence is.</p>
<p><strong>Key takeaway:</strong> run an AI crawler accessibility check on your localized pages. Incomplete or slow-loading country variants may be entirely invisible to AI systems.</p>
<h2 class="wp-block-heading">What Does This Mean for Revenue? A Note on Attribution</h2>
<p>The reality is that clean attribution between AI visibility and revenue doesn’t quite exist yet. For now, AI systems remain a black box for most analytics stacks, and click-through rates on AI answer engines are significantly lower than traditional search.</p>
<p>That said, the directional evidence is building. <a href="https://www.bain.com/about/media-center/press-releases/20252/consumer-reliance-on-ai-search-results-signals-new-era-of-marketing--bain--company-about-80-of-search-users-rely-on-ai-summaries-at-least-40-of-the-time-on-traditional-search-engines-about-60-of-searches-now-end-without-the-user-progressing-to-a/">Bain & Company’s February 2025 research</a> found that 80% of consumers now rely on AI-written results for at least 40% of their searches. A <a href="https://www.bain.com/about/media-center/press-releases/20252/agentic-ai-poised-to-disrupt-retail-even-with-50-of-consumers-cautious-of-fully-autonomous-purchasesbain--company/">separate Bain report from November 2025</a> found that 30–45% of US consumers already use AI specifically for product research and comparison, and that AI now accounts for up to 25% of referral traffic for some retailers. The research and purchase consideration stages are already happening inside AI platforms before buyers ever reach a brand’s site.</p>
<p>And in this specific dataset, the directional data continues: one ecommerce brand saw AI crawler requests more than double from Q1 to Q3 2025. Over the same period, publicly reported earnings increased by 109.7%, accounting for nearly $80M in additional revenue. We’re not claiming causation, but the parallel is consistent with a model where AI visibility amplifies demand rather than directly creating it.</p>
<figure class="wp-block-image size-large"><img loading="lazy" decoding="async" width="1024" height="576" src="https://prerender.io/wp-content/uploads/image-151-1024x576.png" alt="" class="wp-image-7622" srcset="https://prerender.io/wp-content/uploads/image-151-1024x576.png 1024w, https://prerender.io/wp-content/uploads/image-151-300x169.png 300w, https://prerender.io/wp-content/uploads/image-151-768x432.png 768w, https://prerender.io/wp-content/uploads/image-151-1536x864.png 1536w, https://prerender.io/wp-content/uploads/image-151.png 1920w" sizes="(max-width: 1024px) 100vw, 1024px" /></figure>
<p><em>Data source: <a href="http://prerender.io">Prerender.io</a> dashboard of a key enterprise client. </em></p>
<p>The more useful frame: think of AI visibility as revenue protection and a growing acquisition channel, even if it’s not entirely attributable yet. If AI systems retrieve your content incorrectly, partially, or not at all, your brand is excluded from the consideration set before the funnel begins. Buyers arrive at competitors already shaped by AI summaries your brand had no part in.</p>
<h2 class="wp-block-heading">A Measurement Framework</h2>
<p>Given the current attribution gap, here’s a framework for how you can track AI visibility:</p>
<h3 class="wp-block-heading">1. Brand share of voice across AI platforms.</h3>
<p>Track how often your brand is mentioned across ChatGPT, Perplexity, Gemini, and Copilot relative to competitors. This upstream signal predicts downstream revenue influence before it shows up in your analytics. OtterlyAI is the most practical tool we’ve seen for making this measurable.</p>
<h3 class="wp-block-heading">2. Citation monitoring on owned pages. </h3>
<p>This tells you whether your content is being accurately represented. Being cited with incorrect pricing or outdated product details is actively damaging at the moment decisions are being formed. Use Screaming Frog to surface a full URL list, then audit your presence on third-party platforms like G2, Capterra, and Trustpilot.</p>
<h3 class="wp-block-heading">3. AI crawler behavior. </h3>
<p>Measuring AI crawler activity shows whether your site is being actively retrieved by AI systems at all. Growing retrieval volume—like the doubling observed in our ecommerce case—is a directional signal worth tracking alongside business outcomes. You can use a solution like Prerender.io to provide visibility into your AI bot behavior, particularly in comparison to search or social crawlers, and identify whether pages are being served to AI crawlers in a fully rendered state.</p>
<figure class="wp-block-image size-large"><img loading="lazy" decoding="async" width="1024" height="380" src="https://prerender.io/wp-content/uploads/prerender-dashboard-ai-crawler-behavior-1024x380.png" alt="" class="wp-image-7644" srcset="https://prerender.io/wp-content/uploads/prerender-dashboard-ai-crawler-behavior-1024x380.png 1024w, https://prerender.io/wp-content/uploads/prerender-dashboard-ai-crawler-behavior-300x111.png 300w, https://prerender.io/wp-content/uploads/prerender-dashboard-ai-crawler-behavior-768x285.png 768w, https://prerender.io/wp-content/uploads/prerender-dashboard-ai-crawler-behavior-1536x570.png 1536w, https://prerender.io/wp-content/uploads/prerender-dashboard-ai-crawler-behavior-2048x760.png 2048w" sizes="(max-width: 1024px) 100vw, 1024px" /></figure>
<p><em>Prerender.io dashboard of an anonymized client</em></p>
<h3 class="wp-block-heading">4. Branded and direct traffic. </h3>
<p>This captures the indirect effect. Users who encounter your brand in AI-generated answers often don’t click through immediately—they find you later via direct or branded search. Correlating share of voice trends against branded traffic growth gives you a reasonable proxy for AI-driven influence, even without direct attribution.</p>
<figure class="wp-block-image size-large"><img loading="lazy" decoding="async" width="1024" height="576" src="https://prerender.io/wp-content/uploads/image-150-1024x576.png" alt="" class="wp-image-7621" srcset="https://prerender.io/wp-content/uploads/image-150-1024x576.png 1024w, https://prerender.io/wp-content/uploads/image-150-300x169.png 300w, https://prerender.io/wp-content/uploads/image-150-768x432.png 768w, https://prerender.io/wp-content/uploads/image-150-1536x864.png 1536w, https://prerender.io/wp-content/uploads/image-150.png 1920w" sizes="(max-width: 1024px) 100vw, 1024px" /></figure>
<h2 class="wp-block-heading">Summing Up: Three Things That Impact Your Presence in AI Search </h2>
<h3 class="wp-block-heading">1. Technical retrievability is a must-have, not a nice-to-have. </h3>
<p>AI systems can only work with content they can access and read completely. For JavaScript-heavy sites, serving a fully rendered, stable version of each page to AI crawlers isn’t optional. It’s table stakes for everything else.</p>
<h3 class="wp-block-heading">2. Content needs to answer, not just attract. </h3>
<p>The pages AI systems return to are self-contained, clearly structured, and genuinely useful. If a page can’t serve as a complete answer to a specific question on its own, it’s a weak candidate for AI retrieval.</p>
<h3 class="wp-block-heading">3. Third-party presence isn’t optional.</h3>
<p>The majority of AI citations come from outside your owned properties. Brands that earn coverage in the review sites, community platforms, and media that AI systems trust will win disproportionate attribution—regardless of how strong their own content is. Influence without attribution still influences. But it’s more valuable when it’s yours. Ensure that you are prioritizing your third-party presence as a core foundation of your marketing strategy.</p>
<h2 class="wp-block-heading">FAQs</h2>
<h3 class="wp-block-heading">What types of pages does ChatGPT retrieve most often? </h3>
<p>Self-contained pages that answer a specific question without requiring additional navigation: FAQs, buying guides, technical documentation, category pages, and evergreen editorial content.</p>
<h3 class="wp-block-heading">Why is Reddit cited more than brand websites in AI search? </h3>
<p>AI systems are built to prioritize third-party validation and community consensus. Reddit provides peer-generated, question-and-answer content that maps directly onto how AI systems construct responses.</p>
<h3 class="wp-block-heading">Does page speed affect AI search visibility? </h3>
<p>Yes, particularly for ChatGPT. Faster-loading pages are more likely to be included when an AI system has limited time to retrieve and render content.</p>
<h3 class="wp-block-heading">What’s the biggest technical barrier to AI visibility? </h3>
<p>There are a few: bot-blocking configurations and JavaScript rendering. Many enterprise sites inadvertently block AI crawlers with security rules originally written for traditional search engines, resulting in complete invisibility. JavaScript-heavy sites also block AI crawlers from accessing your pages in the first place. Server-side rendering or prerendering with <a href="http://prerender.io">Prerender.io</a> can be a solution here.</p>
<h3 class="wp-block-heading">How do I track whether AI systems are citing my brand? </h3>
<p>Monitor brand share of voice using an AI search analytics tool like <a href="https://otterly.ai/">OtterlyAI</a>. For citation accuracy, combine a Screaming Frog URL crawl with manual audits of third-party review platforms.</p>
<h3 class="wp-block-heading">Does strong SEO automatically translate to strong AI search visibility? </h3>
<p>In many ways, yes, but this isn’t guaranteed. The brands winning in AI search aren’t always the ones with the strongest content, which is a core ranking factor for traditional SEO. They’re the ones whose pages are technically accessible, structured to serve as direct answers, and with a strong third-party presence.</p>
<p></p>
]]></content:encoded>
</item>
<item>
<title>How Social Media and Open Graph Tags Impact LLM Training Data</title>
<link>https://prerender.io/blog/how-open-graph-tags-impact-llm-training-data/</link>
<dc:creator><![CDATA[Prerender Team]]></dc:creator>
<pubDate>Sun, 30 Nov 2025 23:02:28 +0000</pubDate>
<category><![CDATA[AI SEO]]></category>
<category><![CDATA[Prerendering]]></category>
<category><![CDATA[open graph]]></category>
<category><![CDATA[social sharing]]></category>
<guid isPermaLink="false">https://prerender.io/?p=6588</guid>
<description><![CDATA[Explore how Open Graph tags and rich social snippets provide metadata signals for AI. See how optimizing social previews and open graph rendering helps your site stand out in AI search and training datasets.]]></description>
<content:encoded><![CDATA[<p data-block-id="6jo23">Social media isn’t just shaping how people discover your content; it’s shaping how AI understands it. As posts, previews, and metadata flow into large language model (LLM) training datasets, the way your content appears on social platforms becomes part of the foundation that informs how AI systems interpret the web.</p>
<p data-block-id="574eq">In this post, we’ll explore how social media data and <a href="https://prerender.io/blog/benefits-of-using-open-graph/" target="_blank" rel="noopener noreferrer">Open Graph (OG) tags</a> contribute to LLM training, why it matters for AI visibility, and how to optimize your metadata so crawlers and models see your content clearly.</p>
<h2 id="3c5aa" data-block-id="3c5aa">The Role of Social Media in LLM Training</h2>
<p data-block-id="3ukg2">AI models like OpenAI’s GPT, Anthropic’s Claude, and Google’s Gemini learn from vast datasets of public web content, including news, blogs, forums, and social media posts. Social platforms are especially valuable because they reflect how humans communicate in real contexts.</p>
<p data-block-id="3jafk">For instance:</p>
<ul>
<li><strong>Replies and comments</strong> reveal how people respond to ideas.</li>
<li><strong> Likes, shares, and retweets </strong>signal engagement and perceived relevance.</li>
<li><strong> Hashtags and trending topics </strong>show how information spreads and evolves across communities.</li>
</ul>
<p data-block-id="e3590">Meta’s LLaMA 3 offers a clear example of this in action. The <a href="https://medium.com/@mendicott/meta-trains-llama-models-using-public-facebook-and-instagram-data-e173372e03de" target="_blank" rel="noopener noreferrer">company confirmed</a> that the model was trained on publicly shared Facebook and Instagram posts. And according to Mark Zuckerberg, Meta’s corpus of public social data exceeds Common Crawl in size, meaning billions of posts and their interactions helped teach the model.</p>
<p data-block-id="f9lka">With social media, of course, ethical AI training respects privacy and terms of service, so only publicly accessible content can be included. Even with that limitation, it remains one of the richest sources of human expression available to LLMs.</p>
<p data-block-id="7mpic">How your content is represented online, especially through Open Graph tags and other metadata signals for AI, also helps LLMs interpret context more accurately.</p>
<p data-block-id="dc09c"><strong>Related:</strong> <u><a href="https://prerender.io/blog/seo-and-social-media/#elementor-toc__heading-anchor-1" target="_blank" rel="noopener noreferrer">Discover the role of social media in SEO</a>.</u></p>
<h2 id="aojb8" data-block-id="aojb8">Open Graph Tags and Their Relevance to AI Systems</h2>
<p data-block-id="1j648">Facebook originally created Open Graph meta tags to standardize how shared links display across social platforms. Today, they‘re a metadata layer used by LinkedIn, X (formerly Twitter), and other social platforms to generate rich previews that represent your page’s details.</p>
<figure class="image strchf-type-image undefined strchf-size-undefined strchf-align-center"><picture><source srcset="https://images.storychief.com/account_57345/qdifzaaaabkleqvqdanfsovjrfesbaaaaaelftksuqmcc_346cd7f3dceebd2470c35027f8298dd5_800.jpg 1x" media="(max-width: 768px)" /><source srcset="https://images.storychief.com/account_57345/qdifzaaaabkleqvqdanfsovjrfesbaaaaaelftksuqmcc_346cd7f3dceebd2470c35027f8298dd5_800.jpg 1x" media="(min-width: 769px)" /><img decoding="async" src="https://images.storychief.com/account_57345/qdifzaaaabkleqvqdanfsovjrfesbaaaaaelftksuqmcc_346cd7f3dceebd2470c35027f8298dd5_800.jpg" alt="Social metadata used to generate rich previews" /></picture></figure>
<p data-block-id="4jnn7">When someone shares your page link, these meta tags tell the social platform what to display. But beyond improving social experience, OG tags also serve as structured data for AI systems, offering machine-readable information that helps them interpret the <u><a href="https://www.ibm.com/topics/natural-language-processing" target="_blank" rel="noopener noreferrer">semantic meaning</a></u> of your content.</p>
<p data-block-id="8ivl7">This matters because, when large-scale crawlers like <u><a href="https://commoncrawl.org/" target="_blank" rel="noopener noreferrer">Common Crawl</a></u> index the internet, they extract visible text and also capture the full HTML of each page, including metadata in the <head> section. Each tag then becomes a labeled clue that helps identify the page’s topic, context, and intended summary.</p>
<h3 id="u074" data-block-id="u074"><strong>Example: Basic OG Tags</strong></h3>
<p data-block-id="74pih">Here’s what OG tags look like inside your page’s <head> section:</p>
<p data-block-id="53p36"><em><meta property=”og:title” content=”Your Page Title”></em></p>
<p data-block-id="36c7s"><em><meta property=”og:description” content=”A short summary of your page content.”></em></p>
<p data-block-id="5h8eg"><em><meta property=”og:image” content=”https://example.com/image.jpg”></em></p>
<p data-block-id="5gfeg"><em><meta property=”og:url” content=” https://example.com/your-page”></em></p>
<p data-block-id="c3ue0">For example, “og:title” serves as a clean label for what the page is about, while “og:description” offers a concise explanation of its focus. Repeated across millions of pages, these tags contribute to a consistent layer of meaning that LLMs can learn from.</p>
<p data-block-id="158f8">While there’s no public evidence that OG tags are <em>weighted</em> in LLM training, their consistent presence across billions of pages makes them statistically influential. Even if they’re treated as standard HTML elements, their structure reinforces meaning, much like headings, anchor text, or schema markup do.</p>
<p data-block-id="4to5b">In other words, consistent and well-configured metadata improves your link preview and also helps to ensure that your pages are interpreted accurately by the AI systems now shaping online search.</p>
<p data-block-id="9gqm"><strong>Resource:</strong> <u><a href="https://prerender.io/blog/how-to-fix-link-previews/" target="_blank" rel="noopener noreferrer">See how to fix your social sharing link previews</a>.</u></p>
<h2 id="8h7q5" data-block-id="8h7q5">How OG Tags Shape AI Understanding: Interpretation, Categorization, and Recall</h2>
<p data-block-id="4ug6l">Every OG tag adds a layer of meaning that helps AI systems interpret and recall your content accurately. When crawlers capture metadata alongside visible text, these tags provide clean, labeled signals that clarify what your page is about and how it should be represented in training data.</p>
<p data-block-id="114dh">Here’s how each core tag contributes:</p>
<figure class="image strchf-type-image undefined strchf-size-undefined strchf-align-center"><picture><source srcset="https://images.storychief.com/account_57345/okxanwaaaazjrefuawc58rynd81xlqaaaabjru5erkjggg_346cd7f3dceebd2470c35027f8298dd5_800.jpg 1x" media="(max-width: 768px)" /><source srcset="https://images.storychief.com/account_57345/okxanwaaaazjrefuawc58rynd81xlqaaaabjru5erkjggg_346cd7f3dceebd2470c35027f8298dd5_800.jpg 1x" media="(min-width: 769px)" /><img decoding="async" src="https://images.storychief.com/account_57345/okxanwaaaazjrefuawc58rynd81xlqaaaabjru5erkjggg_346cd7f3dceebd2470c35027f8298dd5_800.jpg" alt="How Open Graph Tags Shape AI Understanding" /></picture></figure>
<h3 id="6n2eg" data-block-id="6n2eg">Topic Signal and Consistency: “og:title”</h3>
<p data-block-id="tb5s">Your OG title defines the subject of your content. When it’s clear and consistent across your pages, it strengthens your brand’s topical identity and helps AI models align your content with the right themes over time.</p>
<p data-block-id="b8htn">Vague or inconsistent titles, on the other hand, make it harder for models to categorize your pages reliably and may even blur associations with unrelated topics.</p>
<h3 id="e8o6s" data-block-id="e8o6s">Contextual Framing: “og:description”</h3>
<p data-block-id="2uv7u">This tag provides interpretive context. It conveys tone, intent, and framing that the body text alone may not communicate.</p>
<p data-block-id="3a836">Concise, descriptive summaries help both users and AI systems understand your content’s purpose at a glance, improving how it’s classified and recalled later.</p>
<h3 id="fmfsk" data-block-id="fmfsk">Visual-Semantic Association: “og:image”</h3>
<p data-block-id="8f52c">Multimodal AI models, such as GPT-4, Claude 3, and Gemini, train on image–text pairs. When your visuals appear consistently alongside your written content, they reinforce those associations, helping models recognize your brand or subject matter more accurately in multimodal datasets.</p>
<h3 id="dqto0" data-block-id="dqto0">Canonical Authority: “og:url”</h3>
<p data-block-id="ag13l">This tag ensures all shared versions of a page point back to one canonical URL. It prevents duplication, preserves link equity, and strengthens data consistency across crawlers.</p>
<p data-block-id="3k7sg">While <code>og:url</code> differs from <code><link rel="canonical"></code> (used for SEO), both should match to maintain coherence for both search engines and AI training systems.</p>
<h2 id="brm1h" data-block-id="brm1h">Ensuring Your Metadata is Accessible to LLM Crawlers</h2>
<p data-block-id="dk4rh">Many modern websites, especially <u><a href="https://prerender.io/blog/how-to-optimize-single-page-applications-spas-for-crawling-and-indexing/" target="_blank" rel="noopener noreferrer">Single Page Applications (SPAs)</a></u> built with <u><a href="https://prerender.io/framework/react/" target="_blank" rel="noopener noreferrer">React</a></u>, <u><a href="https://prerender.io/framework/vue-js/" target="_blank" rel="noopener noreferrer">Vue</a></u>, or <u><a href="https://prerender.io/blog/angularjs-pros-cons-and-optimization/" target="_blank" rel="noopener noreferrer">Angular</a></u>, render content dynamically.</p>
<p data-block-id="22b2u">Unfortunately, most <u><a href="https://prerender.io/blog/understanding-web-crawlers-traditional-ai/" target="_blank" rel="noopener noreferrer">social bots and AI crawlers</a></u> can’t execute JavaScript, which makes it impossible for them to access your Open Graph tags. That means your intended web content assets might not load before a crawler takes its snapshot, leading to broken previews and missing metadata signals.</p>
<p data-block-id="d0n3q">The typical fix is <a href="https://prerender.io/blog/what-is-srr-and-why-do-you-need-to-know/" target="_blank" rel="noopener noreferrer">server-side rendering (SSR),</a> but it’s complex and costly at scale. That’s why many teams rely on <strong>Prerender.io</strong>. This open graph rendering solution delivers fully rendered HTML versions of your pages, ensuring all bots and crawlers can access your OG metadata.</p>
<p data-block-id="e2aoa"><u><a href="http://prerender.io/" target="_blank" rel="noopener noreferrer">Prerender.io</a></u> is a tool built to render your web JavaScript files into their HTML versions in advance, giving social media bots and crawlers unrestricted access to your OG tags and social metadata.</p>
<p data-block-id="cvcfb">Here’s how it works:</p>
<ol type="1">
<li><strong>Prerender.io detects crawlers </strong>by their user-agent string.</li>
<li><strong>It renders your page in a headless browser</strong>, captures the full HTML output (including OG tags), and caches the result.</li>
<li>Crawlers then receive a <strong>clean, prerendered HTML snapshot</strong>.</li>
</ol>
<p data-block-id="f5dhk">In addition, Prerender.io’s dynamic caching system automatically updates snapshots whenever your content changes, keeping your rich social snippets accurate while saving <u><a href="https://prerender.io/blog/crawl-budget-management-for-large-websites/" target="_blank" rel="noopener noreferrer">crawl budget</a></u>.</p>
<p data-block-id="eg0cs">Curious to learn more about Prerender.io and the complete <u><a href="https://prerender.io/blog/a-guide-to-prerender-process-and-benefits/" target="_blank" rel="noopener noreferrer">AI SEO benefits</a></u> it offers? Watch this video to learn more.</p>
<figure class="video strchf-type-video regular strchf-size-regular strchf-align-center">
<div class="embed-container">
<div style="max-width: 100%; position: relative; padding-top: 56.5%;"><iframe style="position: absolute; top: 0px; left: 0px; width: 100%; height: 100%; border: none;" title="How Does Prerender.io Work? A Quick Explainer" src="https://www.youtube-nocookie.com/embed/OxNt36HhCP4?feature=oembed" width="200" height="113" frameborder="0" allowfullscreen="allowfullscreen"></iframe></div>
</div>
</figure>
<h2 id="41g98" data-block-id="41g98">Best Practices for Optimizing Your Social Markup for AI Discoverability</h2>
<p data-block-id="1qd1p">To make your content more accessible to both social platforms and AI crawlers, follow a few practical steps when setting up and maintaining your Open Graph metadata. </p>
<figure class="image strchf-type-image undefined strchf-size-undefined strchf-align-center"><picture><source srcset="https://images.storychief.com/account_57345/ngaaaaaelftksuqmcc_346cd7f3dceebd2470c35027f8298dd5_800.jpg 1x" media="(max-width: 768px)" /><source srcset="https://images.storychief.com/account_57345/ngaaaaaelftksuqmcc_346cd7f3dceebd2470c35027f8298dd5_800.jpg 1x" media="(min-width: 769px)" /><img decoding="async" src="https://images.storychief.com/account_57345/ngaaaaaelftksuqmcc_346cd7f3dceebd2470c35027f8298dd5_800.jpg" alt="Optimizing social previews for LLM visibility" /></picture></figure>
<p data-block-id="7j9v9">Here’s how to optimize social previews for LLM visibility:</p>
<h3 id="9tubo" data-block-id="9tubo">1. Include Core OG Tags</h3>
<p data-block-id="apdmi">Use the essential OG tags: <code>og:title</code>, <code>og:description</code>, <code>og:image</code>, and <code>og:url</code>, on every key page. Keep titles concise (40–60 characters) and descriptions specific (100–160 characters) to define your page’s core identity clearly.</p>
<h3 id="9miuv" data-block-id="9miuv">2. Use High-Resolution Images</h3>
<p data-block-id="9rc0j">Select a clear, relevant image at least 1200×630 px for <code>og:image</code>. Strong visuals not only capture attention on social feeds but also help multimodal AI systems link your content with consistent visual context.</p>
<h3 id="7kdfc" data-block-id="7kdfc">3. Add Descriptive Alt Text</h3>
<p data-block-id="3ugab">Even though OG tags don’t contain alt attributes, surrounding image text and filenames inform AI models about context. So, whenever possible, make sure your image(s) have meaningful <u><a href="https://prerender.io/blog/alt-text-for-images-best-practices/" target="_blank" rel="noopener noreferrer">alt text</a></u>.</p>
<h3 id="ao9hn" data-block-id="ao9hn">4. Keep Metadata Consistent and Up to Date</h3>
<p data-block-id="974lo">Align your OG metadata with your page’s <code><title></code> and visible content. If you update a headline or featured image, refresh your OG tags as well. Consistency helps crawlers and AI systems interpret your content correctly.</p>
<h3 id="9et75" data-block-id="9et75">5. Ensure Crawler Access and Rendering</h3>
<p data-block-id="f5jf9">Verify that your robots.txt file or server headers don’t block essential user agents such as Twitterbot, LinkedInBot, FacebookExternalHit, or GPTBot. If your site relies on JavaScript, use server-side rendering or a prerendering solution (like Prerender.io) to ensure crawlers can read your full HTML snapshots and metadata.</p>
<h3 id="fve7j" data-block-id="fve7j">6. Add Platform-Specific Tags</h3>
<p data-block-id="cs2t7">While OG tags are widely supported, platforms like X use their own metadata markup (twitter:card, twitter:title, etc.). If these tags are missing or incorrect, it can result in issues such as your Twitter link preview not showing. Including the correct <a href="https://prerender.io/blog/how-to-use-twitter-cards-for-content-amplification/" target="_blank" rel="noopener noreferrer">platform-specific tags</a> ensures consistent link rendering across social platforms and helps AI crawlers accurately interpret structured context from each platform.</p>
<h3 id="ekk8q" data-block-id="ekk8q">7. Implement Structured Data</h3>
<p data-block-id="blj2g">Complement OG tags with JSON-LD schemas (e.g., <code>Article</code>, <code>Product</code>, <code>Organization</code>). Structured data helps define your page’s entities and attributes, making it easier for both search engines and AI systems to understand your content.</p>
<p data-block-id="elkhf"><strong>Resource:</strong> <u><a href="https://prerender.io/blog/5-types-of-schema-markup-dynamic-websites-should-implement-including-a-tutorial/" target="_blank" rel="noopener noreferrer">Learn more about different types of schema markups and how to apply them</a></u>.</p>
<h3 id="cigtb" data-block-id="cigtb">8. Test Your Social Previews</h3>
<p data-block-id="pd6v">Use tools like <u><a href="https://developers.facebook.com/tools/debug/" target="_blank" rel="noopener noreferrer">Facebook Sharing Debugger</a></u>, <u><a href="https://www.linkedin.com/post-inspector/" target="_blank" rel="noopener noreferrer">LinkedIn Post Inspector</a></u>, or <u><a href="https://threadcreator.com/tools/twitter-card-validator" target="_blank" rel="noopener noreferrer">Twitter Card Validator</a></u> to verify live previews and detect any rendering issues early.</p>
<h2 id="abhrk" data-block-id="abhrk">Optimizing Open Graph Tags for AI Search and Rich Social Snippets with Prerender.io</h2>
<p data-block-id="7407j">Your Open Graph meta tags do more than create attractive link previews; they also act as structured data for AI. Every tag, image, and description adds metadata signals that help LLMs interpret your content’s meaning, category, and visual context.</p>
<p data-block-id="epkg1">When properly configured, these tags serve both audiences at once:</p>
<ul>
<li><strong>For humans</strong>, they create rich social snippets that improve engagement and click-through rates.</li>
<li><strong>For AI systems</strong>, they clarify how your content should be indexed, recalled, and represented within training datasets.</li>
</ul>
<p data-block-id="6n4ga">The takeaway: optimizing your social metadata is no longer just about presentation—it’s about discoverability. By keeping your tags consistent, crawlable, and accurately rendered, you ensure that your website speaks the same language as both social platforms and AI systems.</p>
<p data-block-id="ed5r9">Prerender.io simplifies this process with automated open graph rendering, ensuring your OG tags and structured data are visible to every crawler, including AI bots.</p>
<p data-block-id="2j1rj">Ready to make sure your metadata is visible everywhere it matters? <a href="https://prerender.io/pricing/" target="_blank" rel="noopener noreferrer"><strong>Get</strong> <strong>started with Prerender.io</strong></a><strong>.</strong></p>
<h2 id="am51b" data-block-id="am51b">FAQs About Open Graph Tags and AI Visibility</h2>
<h3 id="3mhef" data-block-id="3mhef"><strong>1. What Are Open Graph Tags and Why Do They Matter for AI Discoverability?</strong></h3>
<p data-block-id="75g1f">Open Graph tags define how your content appears when shared on social media. They also serve as machine-readable labels that help <a href="https://prerender.io/blog/understanding-web-crawlers-traditional-ai/" target="_blank" rel="noopener noreferrer">AI crawlers</a> interpret your page’s topic, intent, and relationships when collecting LLM training data.</p>
<h3 id="207j" data-block-id="207j"><strong>2. How Do Open Graph Meta Tags Improve Rich Social Snippets?</strong></h3>
<p data-block-id="n9b6">OG tags tell platforms exactly which title, description, and image to display, creating rich social snippets that attract clicks. This consistency also reinforces accurate metadata signals for AI models learning from public web content.</p>
<h3 id="8h9l1" data-block-id="8h9l1"><strong>3. Why Isn’t My Twitter Link Preview Showing Correctly?</strong></h3>
<p data-block-id="38kfs">If your Twitter link preview isn’t showing, check that you’ve added the right <code>twitter:card</code> and <code>twitter:title</code> tags alongside your OG tags. Use the <a href="https://devcommunity.x.com/t/twitter-card-validation-tool/149666" target="_blank" rel="noopener noreferrer">Twitter Card Validator </a>to test and debug your social metadata.</p>
<h3 id="5ma6p" data-block-id="5ma6p"><strong>4. How Can Open Graph Rendering Affect AI Training Signals?</strong></h3>
<p data-block-id="f1sou">Incomplete or missing OG tags can cause AI crawlers to capture fragmented data, weakening training signals. Using a <a href="https://prerender.io/ai-search/" target="_blank" rel="noopener noreferrer">prerendering solution</a> ensures your metadata loads as clean HTML before crawlers or social bots take their snapshots.</p>
<h3 id="4p9q1" data-block-id="4p9q1"><strong>5. How Do I Optimize My Website for AI Search?</strong></h3>
<p data-block-id="cepof">Combine strong Open Graph tags with structured data for AI (e.g., JSON-LD schemas). Ensure that all metadata is consistent, accessible to crawlers, and kept up to date. This approach improves both search engine indexing and <a href="https://prerender.io/blog/ai-optimization-technical-seo-guide/" target="_blank" rel="noopener noreferrer">AI discoverability</a>.</p>
<p><!-- strchf script --><script>if(window.strchfSettings === undefined) window.strchfSettings = {};window.strchfSettings.stats = {url: "https://prerender.storychief.io/en/how-open-graph-tags-impact-llm-training-data?id=129959776&type=2",title: "How Social Media and Open Graph Tags Impact LLM Training Data",siteId: "20338",id: ""};(function(d, s, id) {var js, sjs = d.getElementsByTagName(s)[0];if (d.getElementById(id)) {window.strchf.update(); return;}js = d.createElement(s); js.id = id;js.src = "https://d37oebn0w9ir6a.cloudfront.net/scripts/v0/strchf.js";js.async = true;sjs.parentNode.insertBefore(js, sjs);}(document, 'script', 'storychief-jssdk'))</script><!-- End strchf script --></p>
]]></content:encoded>
</item>
<item>
<title>“AI: Evolve or Get Left Behind” – Practical Tips for AI Tool Adoption with Jonathan Bentz</title>
<link>https://prerender.io/blog/tips-for-ai-tool-adoption-get-discovered-podcast/</link>
<dc:creator><![CDATA[Prerender]]></dc:creator>
<pubDate>Tue, 21 Oct 2025 13:28:05 +0000</pubDate>
<category><![CDATA[Podcast]]></category>
<category><![CDATA[ai podcast]]></category>
<category><![CDATA[get discovered]]></category>
<category><![CDATA[get discovered podcast]]></category>
<category><![CDATA[podcast episodes]]></category>
<guid isPermaLink="false">https://prerender.io/?p=6283</guid>
<description><![CDATA[Jonathan Bentz from Direct Online Marketing shares beginner-friendly tips for AI adoption.]]></description>
<content:encoded><![CDATA[
<p>In the fourth episode of Get Discovered, a podcast by Prerender.io, host Joe Walsh talks with Jonathan Bentz (AKA “Bentz”) from <a href="https://prerender.io/resources/case-studies/award-winning-podcast-studio-dom/">Direct Online Marketing (DOM)</a>. In this episode, the two discuss how anyone can approach AI adoption strategically—identifying where AI adds real value, where it still struggles, and how teams can integrate it into daily workflows without losing the human edge.</p>
<p>Whether you’re just starting out with AI or you’re an AI expert, this episode shares thoughtful, practical tips on how to adopt any AI tool. Watch the full episode below or read on for our full summary.</p>
<figure class="wp-block-embed is-type-video is-provider-youtube wp-block-embed-youtube wp-embed-aspect-16-9 wp-has-aspect-ratio"><div class="wp-block-embed__wrapper">
<iframe title=""AI: Evolve or Get Left Behind - Tips for AI Tool Adoption with Jonathan Bentz from @Directom" width="640" height="360" src="https://www.youtube.com/embed/WixSy0PZZSo?feature=oembed" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>
</div></figure>
<h2 class="wp-block-heading">Understanding AI and Its Role in Modern Work</h2>
<p>AI isn’t a future concept anymore: it’s already (deeply) woven into the systems and tools you use daily. The real question isn’t <em>whether</em> to use AI but <em>how</em> to evolve with it. </p>
<p>And sometimes, that means going back to basics, such as understanding what AI really is. When we think of how we may define AI, we can arrive at a few key examples in our day-to-day lives, says Jonathan. He defines AI, at its core, as any nonhuman system capable of making decisions. This goes well beyond ChatGPT, such as: </p>
<ul class="wp-block-list">
<li>Voice assistants like Siri or Alexa</li>
<li>Car systems that adapt for safety and navigation</li>
<li>Factory robots managing repetitive tasks</li>
<li>Large language models that generate natural responses</li>
</ul>
<p>All these tools process data and act independently, enhancing human efficiency rather than replacing it. But it’s important to ensure we’re using AI tools when they have the most impact.</p>
<h2 class="wp-block-heading">Where AI Delivers Value (and Where It Doesn’t)</h2>
<p>Jonathan and Joe talk about where AI truly delivers value. For Jonathan, AI excels at handling structured, repetitive, or templated tasks that eat up time but don’t require deep expertise. This could include:</p>
<ul class="wp-block-list">
<li>Brainstorming ideas or refining concepts</li>
<li>Drafting SOPs, documentation, and templates</li>
<li>Turning sketches into usable visuals</li>
<li>Writing or editing clear, professional emails</li>
</ul>
<p><em>Further reading: <a href="https://prerender.io/blog/how-to-get-indexed-on-ai-platforms/">How to Get Indexed on AI Platforms</a></em></p>
<p>But he certainly doesn’t rely on AI for everything. He limits his AI usage to a few other areas:</p>
<ul class="wp-block-list">
<li>Complex technical or domain-specific work</li>
<li>Context-heavy tasks like transforming long documents into web pages</li>
<li>Unique, judgment-based situations that require human oversight</li>
</ul>
<p>Jonathan’s tip? <strong>Think of AI as the system that covers the first 80 percent, so you can focus on the remaining 20 percent </strong>(the creative and strategic parts that still demand a human touch).</p>
<h2 class="wp-block-heading">How to Adopt AI Tools Thoughtfully</h2>
<p>Integrating AI doesn’t have to be overwhelming. Success comes from experimentation, structure, and validation. Here are some practical adoption tips that Jonathan recommends.</p>
<h3 class="wp-block-heading">10 Practical AI Adoption Tips</h3>
<ol class="wp-block-list">
<li><strong>Experiment with multiple tools and compare results.</strong></li>
<li><strong>Start small with controlled, low-risk prompts.</strong></li>
<li><strong>Assign roles in chat threads for better context retention.</strong></li>
<li><strong>Use AI for 80 percent of the work, then refine it manually.</strong></li>
<li><strong>Cross-check results across tools for accuracy.</strong></li>
<li><strong>Stay privacy-conscious when entering proprietary data.</strong></li>
<li><strong>Automate repetitive workflows using built-in actions.</strong></li>
<li><strong>Track where your human input adds value.</strong></li>
<li><strong>Train your team to integrate AI into documentation and onboarding.</strong></li>
<li><strong>Keep communication human, even when using automation.</strong></li>
</ol>
<p>These tips will help anyone who’s just getting started with AI to experiment, become more efficient, and create better work. </p>
<p>Jonathan also shares tips for prompting, too. Here are his suggestions for LLM prompting:</p>
<ol class="wp-block-list">
<li>Define a clear role or task for each session (“You are my content editor”).</li>
<li>Keep long-term conversations open so the model learns context.</li>
<li>Iterate prompts instead of starting from scratch.</li>
</ol>
<p>From there, you’ll get higher-quality outputs with less time spent on reviews, edits, and revisions. </p>
<h2 class="wp-block-heading">How to Navigate AI Risks and Hype</h2>
<p>However, as AI adoption grows, it’s important to separate genuine innovation from marketing noise. Jonathan and Joe have a few key recommendations for how you can use LLMs thoughtfully—and when you should steer clear of relying on it.</p>
<ul class="wp-block-list">
<li><strong>Stay cautious—especially with YMYL topics.</strong> In sensitive areas like health, finance, or legal advice (<em>Your Money, Your Life</em>), always verify outputs manually. Treat AI as an assistant, not a final authority.
<ul class="wp-block-list">
<li><em>Further reading: <a href="https://prerender.io/blog/anna-nadeina-saas-group/">How to be Cautiously Optimistic About AI: Anna Nadeina from saas.group</a></em><br></li>
</ul>
</li>
<li><strong>Watch for “AI lipstick.”</strong> Many tools labelled “AI-powered” are simply repackaged automations. Evaluate critically. Ask whether a tool truly learns and adapts or just executes preset workflows.<br></li>
<li><strong>Understand commercial incentives.</strong> AI tools are capital-intensive to build, and monetization can shape results. Stay aware of how advertising or commercial bias might influence generated recommendations.</li>
</ul>
<h2 class="wp-block-heading">How to Build the Right Mindset for the AI Era</h2>
<p>At the end of the day, AI is (definitely) here to stay. And adopting it successfully is about skills, habits, and adaptability. Here are some tips and mindset shifts that Jonathan recommends for any SEO or marketer on how to start integrating AI in your workflow.</p>
<ul class="wp-block-list">
<li><strong>Experimentation:</strong> Learn by testing and iterating.</li>
<li><strong>Critical thinking:</strong> Validate important information before acting.</li>
<li><strong>Adaptability:</strong> Update workflows as AI capabilities evolve.</li>
<li><strong>Strategic focus:</strong> Prioritize human judgment and creativity.</li>
<li><strong>Continuous learning:</strong> Adjust and refine as new tools emerge.</li>
</ul>
<h3 class="wp-block-heading">A Quick-Start AI Checklist</h3>
<p>Ready to kick off your optimal AI usage? Here’s a quick-start checklist: </p>
<ul class="wp-block-list">
<li>Test 2–5 AI tools with the same prompt and compare outputs.</li>
<li>Create ongoing chat threads with defined roles.</li>
<li>Identify one repetitive task to automate this week.</li>
<li>Track edits to capture your team’s added value.</li>
<li>Define human review points for high-stakes topics.</li>
</ul>
<h2 class="wp-block-heading">Key Takeaways: Start Small, Test Often and Stay Curious</h2>
<p>AI won’t replace thoughtful professionals, says Jonathan. His prediction? It will simply amplify efficiency and change what “high-value work” really means. The competitive advantage belongs to those who adapt quickly, validate consistently, and use AI as a force multiplier.</p>
<p>Key takeaways from Jonathan’s episode on how to adopt AI tools so that we evolve—not get left behind. Start small. Test often. Stay curious. The more you learn to integrate AI intentionally, the more it enhances, rather than replaces, your expertise.</p>
<h3 class="wp-block-heading">About This Episode</h3>
<p>This article is based on a conversation from <em>Get Discovered</em>, a podcast by Prerender.io on AI, SEO, and online discoverability. Listen to the episode on <a href="https://open.spotify.com/episode/4MQ2lShwCD02Dsdd2q1swA?si=SOlZrZhRSTy0sB1uRCr3YQ">Spotify</a>, <a href="https://podcasts.apple.com/us/podcast/ai-an-evolve-or-get-left-behind-situation-jonathan/id1840604392?i=1000732821422">Apple Podcasts</a>, or <a href="https://youtu.be/WixSy0PZZSo">YouTube</a>. If you’d like to be a guest on future seasons of the podcast, <a href="https://prerender.io/podcast/">reach out</a>.</p>
]]></content:encoded>
</item>
<item>
<title>Understanding Web Crawlers: Traditional vs. OpenAI’s Bots</title>
<link>https://prerender.io/blog/understanding-web-crawlers-traditional-ai/</link>
<dc:creator><![CDATA[Prerender]]></dc:creator>
<pubDate>Thu, 25 Sep 2025 09:00:37 +0000</pubDate>
<category><![CDATA[AI SEO]]></category>
<category><![CDATA[AI crawlers]]></category>
<category><![CDATA[prerendering]]></category>
<guid isPermaLink="false">https://prerender.io/?p=6308</guid>
<description><![CDATA[AI crawlers often miss JavaScript content, making parts of your site invisible. Learn the key differences between traditional crawlers and AI crawlers, and how prerendering can boost your AI visibility.]]></description>
<content:encoded><![CDATA[
<p>AI platforms like ChatGPT are transforming the search landscape. But unfortunately, there’s a good chance that <strong>your website’s important information might be completely invisible to AI crawlers</strong>.</p>
<p>And if your content isn’t accessible to AI crawlers, it won’t show up in answers from AI platforms like ChatGPT, Perplexity, or Claude.</p>
<p>Whether your content is visible or not largely depends on JavaScript, a website framework that makes beautiful websites, but is hard for bots and crawlers to access efficiently.</p>
<p>So, what are the differences between ‘traditional’ crawlers and AI ones? What content does JavaScript impact most? How do you optimize your content for AI crawlers, and how can solutions like prerendering help? </p>
<p>Let’s dive into all these questions and answers. Read on for details.</p>
<h2 class="wp-block-heading">What are the Differences Between Traditional and AI Crawlers?</h2>
<p>Traditional crawlers like Googlebot crawl, index, and render website content. Even though Google’s crawlers have their own JavaScript limitations, they can still render JavaScript. But AI crawlers, on the other hand, <strong>won’t execute JavaScript at all</strong>.</p>
<p>Furthermore, AI crawlers still largely rely on what’s available in SERPs to feed their database. Since traditional crawlers can struggle to index JavaScript content, there’s a good chance that this content is hidden from AI crawlers, too.</p>
<p>AI crawlers primarily focus on static HTML websites and prefer clear, structured, and well-formatted text-based content. Your site’s interactive JS elements—such as pop-ups, interactive charts or maps, infinite scroll content, or content hidden behind clickable tabs—might be completely blocked to AI crawlers. They also come with extra limitations:</p>
<ul class="wp-block-list">
<li>OpenAI’s <strong><a href="https://seo.ai/blog/does-chatgpt-and-ai-crawlers-read-javascript" target="_blank" rel="noreferrer noopener nofollow">GPTbot only focuses on and parses the raw HTML content</a></strong> it sees on the initial page load. This means your content won’t be seen if it’s injected via JavaScript.</li>
<li>To ensure efficient scalability,<strong> AI crawlers impose resource constraints and <a href="https://searchengineland.com/ai-optimization-how-to-optimize-your-content-for-ai-search-and-agents-451287#:~:text=Many%20AI%20systems%20have%20tight%20timeouts%20(1%2D5%20seconds)%20for%20retrieving%20content.%C2%A0" target="_blank" rel="noreferrer noopener nofollow">tight timeouts (1-5 seconds)</a></strong>. If your web pages load too slowly, the crawlers might skip them altogether.</li>
<li>Major AI crawlers <strong><a href="https://seo.ai/blog/does-chatgpt-and-ai-crawlers-read-javascript#:~:text=Dynamic%20elements%20generated%20by%20JavaScript%20are%20ignored%20at%20this%20stage." target="_blank" rel="noreferrer noopener nofollow">don’t execute JavaScript</a></strong>, making <strong>dynamic content, such as user reviews, product listings, and lazy-loading sections</strong> on SPAs and JavaScript-heavy websites <strong>nearly invisible</strong>.<br></li>
</ul>
<figure class="wp-block-table"><table class="has-fixed-layout"><thead><tr><th>Features</th><th>Traditional bots</th><th>AI bots</th></tr></thead><tbody><tr><td>Primary purpose</td><td>Indexing web pages for search engine ranking results</td><td>Collecting data to train and improve LLMs and AI models’ performance to generate human-like text responses using NLP and ML models</td></tr><tr><td>Crawling and indexing</td><td>Supports deep and recursive crawling, storing a vast database of pages to index and rank relevant results</td><td>Supports targeted, selective, and context-aware indexing, prioritizing content relevance over volume</td></tr><tr><td>Page rendering</td><td>Supports dynamic web page rendering, with limitations</td><td>Focuses on static HTML content scraping</td></tr><tr><td>JavaScript execution</td><td>Yes, with limitations</td><td>No</td></tr><tr><td>Crawling frequency</td><td>Regular, depending on the website’s authority and popularity. Higher authority websites get crawled more often</td><td>Exhibits more frequent and aggressive crawling patterns, mostly in shorter bursts than traditional crawlers</td></tr><tr><td>Robots.txt support</td><td>Respects and follows instructions in the robots.txt files</td><td>Some AI crawlers may ignore robots.txt files; however, OpenAI’s bots follow robots.txt</td></tr><tr><td>SEO relevance</td><td>It has a direct impact on search ranking and visibility</td><td>Doesn’t directly impact search engine ranking</td></tr></tbody></table></figure>
<p>This difference in both crawlers’ crawling capabilities makes AI crawlers see a stripped-down version of your website. This is especially applicable for websites built on the Angular, React, or Vue frameworks. That’s because these frameworks often rely on Client-Side Rendering (CSR), where most of the page’s content loads on the user’s browser using JavaScript, after it’s initially served.</p>
<p>And since AI crawlers don’t typically wait for JavaScript to load, they only scan the raw HTML available at the first load. <strong>If your page’s key content isn’t present in the HTML and appears only after client-side scripts run, it won’t be visible to the AI crawlers at all.</strong></p>
<p>This is where prerendering or server-side rendering helps ensure that fully-rendered HTML is accessible to AI crawlers, improving its chances of appearing on AI results for users.</p>
<p><strong>Learn more</strong> about the <a href="https://prerender.io/blog/other-rendering-options-vs-prerendering/">difference between prerendering and other rendering options</a>.</p>
<h2 class="wp-block-heading">What Do We Know About OpenAI’s Crawlers?</h2>
<p>ChatGPT is, by far, one the most popular AI search tools. As of July 2025, it currently receives <a href="https://explodingtopics.com/blog/chatgpt-users" target="_blank" rel="noreferrer noopener nofollow">over four billion visits per month</a>.</p>
<p>But OpenAI has a few different crawlers that navigate the web, and it’s important to be aware of each ones’ strengths and limitations. (Note: Oncrawl’s <a href="https://www.oncrawl.com/products/seo-log-analyzer/" target="_blank" rel="noreferrer noopener nofollow">Log Analyzer</a> can help you track this.)</p>
<p>Here’s a high-level overview of each of the three major OpenAI crawlers:</p>
<ul class="wp-block-list">
<li><strong>GPTBot:</strong> This is an offline and asynchronous bot that crawls your website to collect information and train AI and language models, thereby improving AI results. This means that while your website is used to train AI models, <strong>it doesn’t necessarily rank in GPT search results.</strong></li>
<li><strong>ChatGPT-User:</strong> This bot indicates that a real user query made ChatGPT crawl your website in real time to fetch up-to-date content. <strong>Requests from this bot indicate the best signal of visibility.</strong> Ensuring these pages are accessible is crucial for boosting visibility in ChatGPT’s results.</li>
<li><strong>OAI-SearchBot:</strong> This is the indexing bot that runs asynchronously to augment and refresh search results from Bing and other sources.</li>
</ul>
<p>And the crawl frequency of OpenAI’s bots differs significantly from traditional bots. While Google’s bots crawl web pages more frequently depending on content freshness, update history, and authority, <strong>GPTBot has an infrequent or broad crawl frequency</strong> with long revisit intervals for a page. Unless a page is of high value and authority, it may crawl a page once in a few weeks.</p>
<p><strong>OAI-SearchBot</strong>, on the other hand, has a periodic but limited and very infrequent crawl frequency than traditional bots. Similarly, <strong>ChatGPT-User is triggered by a user prompt</strong>, immediately crawling a bunch of URLs upon user requests. This means it doesn’t continuously crawl URLs like traditional bots.</p>
<p>Unlike traditional bots, the crawl budget and volume of OpenAI bots are quite low, selective, and quality-driven, maximizing data quality and prioritizing clean and well-structured content.</p>
<p>Understanding how each of these crawlers operates can help you build out a stronger SEO strategy. And ultimately, <strong><a href="https://www.oncrawl.com/ebooks/ai-evolution-search/" target="_blank" rel="noreferrer noopener nofollow">Oncrawl’s analysis</a> confirms that <em>none </em>of the OpenAI crawlers execute JavaScript.</strong> Despite downloading .js files, OpenAI bots don’t run them.</p>
<p>As a result, without rendering support, your .js content likely won’t be crawlable by the world’s most popular LLM, ChatGPT. And this can have knock-on effects on your bottom line.</p>
<h2 class="wp-block-heading">The Impact of Poor JavaScript Rendering on Businesses</h2>
<p>This limited visibility can harm your website’s performance, but not all parts of your website are affected equally. Here are the elements to be mindful of:</p>
<ul class="wp-block-list">
<li><strong>Dynamic product information,</strong> such as product availability, prices, variants (color/sizes), and discounts.</li>
<li><strong>Lazy-loaded content,</strong> like images, reviews, testimonials, and comments.</li>
<li><strong>Interactive elements,</strong> such as carousels, tabs, modals, and sliders.</li>
<li><strong>Client-side rendered text</strong> is often missed by AI crawlers, as they get a blank page instead of a fully rendered page.</li>
</ul>
<p>Here’s a chart to help keep track of this at a glance:</p>
<figure class="wp-block-image size-large"><img loading="lazy" decoding="async" width="1024" height="494" src="https://prerender.io/wp-content/uploads/Understanding-AI-Web-Crawlers-03-1-1024x494.png" alt="" class="wp-image-6311" srcset="https://prerender.io/wp-content/uploads/Understanding-AI-Web-Crawlers-03-1-1024x494.png 1024w, https://prerender.io/wp-content/uploads/Understanding-AI-Web-Crawlers-03-1-300x145.png 300w, https://prerender.io/wp-content/uploads/Understanding-AI-Web-Crawlers-03-1-768x371.png 768w, https://prerender.io/wp-content/uploads/Understanding-AI-Web-Crawlers-03-1.png 1500w" sizes="(max-width: 1024px) 100vw, 1024px" /></figure>
<h3 class="wp-block-heading"><br>How Poor JavaScript Rendering Can Hurt Your Bottom Line</h3>
<p>Any of this hidden JavaScript content can impact your business, particularly large websites, content-heavy sites with frequent updates, or ecommerce businesses with dynamic content.</p>
<p>For example, ecommerce websites that rely on dynamic product details or quick-changing inventory have the most issues with this. And SaaS brands like <a href="https://neilpatel.com/blog/common-javascript-rendering-issues/" target="_blank" rel="noreferrer noopener nofollow">Asana suffer from content gap issues</a>, as they utilize JavaScript for interactive features and elements across their websites.</p>
<p>This issue of the JavaScript blind spot has <strong>major repercussions for brands and businesses</strong>, the most significant one being <strong>lost visibility, missing content, and as a result, a decline in traffic,</strong> especially with the growing popularity of AI search results and AI Overviews.</p>
<p>Besides the loss of visibility, JS visibility issues can also result in an <strong>incorrect or incomplete representation of your website’s content</strong>, like product details or missing product prices, resulting in customer distrust.</p>
<p>So, how can you solve this?</p>
<h2 class="wp-block-heading">How Prerendering Can Fix Your AI Visibility Issues</h2>
<p>If you’re worried that your content is invisible to AI crawlers because of JavaScript, that’s where prerendering can help.</p>
<h3 class="wp-block-heading">How Prerendering Works</h3>
<p>Prerendering solutions like <a href="https://prerender.io">Prerender.io</a> <strong>generate a fully rendered, static version of your web pages</strong> <em>before </em>bots or crawlers request them. This means that they’re ready to go when an AI crawler visits your page—both traditional and AI crawlers can process HTML much more efficiently.</p>
<p>This rendered HTML version comprises all the essential elements, including the text, images, metadata, dynamic content, videos, and links, without requiring the bots to rely on JavaScript rendering. This means that your important information previously hidden behind JavaScript doesn’t get missed.</p>
<p>Since AI crawlers like OAI-SearchBot and GPTBot have limited JavaScript capabilities and tight timeouts, they may completely skip JS-heavy, complex and slow pages. This is where prerendering can add an extra advantage for you:</p>
<ul class="wp-block-list">
<li><strong>AI crawlers don’t need to wait for scripts to run.</strong></li>
<li><strong>Prerendering provides full context and elements of the page in the initial HTML response.</strong></li>
<li><strong>Crawlers don’t need to rely on dynamic JavaScript rendering to ensure visibility.</strong></li>
</ul>
<p>Especially when you use tools like Prerender.io that enhance your website’s rich snippets and structured data, you can ensure better AI crawlability and faster indexing of your website.</p>
<p>This can have a host of indirect benefits often caused by JavaScript rendering issues, such as slow or broken link previews.</p>
<figure class="wp-block-image size-large"><img loading="lazy" decoding="async" width="1024" height="737" src="https://prerender.io/wp-content/uploads/Whatsapp-share-preview-1024x737.png" alt="" class="wp-image-3640" srcset="https://prerender.io/wp-content/uploads/Whatsapp-share-preview-1024x737.png 1024w, https://prerender.io/wp-content/uploads/Whatsapp-share-preview-300x216.png 300w, https://prerender.io/wp-content/uploads/Whatsapp-share-preview-768x553.png 768w, https://prerender.io/wp-content/uploads/Whatsapp-share-preview.png 1324w" sizes="(max-width: 1024px) 100vw, 1024px" /></figure>
<p><br>Server-side rendering is also a solution to address JS visibility, but there are some downsides. Maintaining in-house SSR requires significant developer resources and ongoing maintenance costs, often costing you much more time and money than prerendering would.</p>
<p>Tools like <a href="https://prerender.io">Prerender.io</a> offer a one-time integration that you can set and forget, ensuring that search engines, AI platforms, and social media platforms swiftly access your content without much oversight.</p>
<p>Here’s a short video of how Prerender.io works:</p>
<figure class="wp-block-embed is-type-video is-provider-youtube wp-block-embed-youtube wp-embed-aspect-16-9 wp-has-aspect-ratio"><div class="wp-block-embed__wrapper">
<iframe title="How Does Prerender.io Work? A Quick Explainer" width="640" height="360" src="https://www.youtube.com/embed/OxNt36HhCP4?feature=oembed" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>
</div></figure>
<h2 class="wp-block-heading">How to “Rank” for AI Crawlers with Prerendering</h2>
<p>Here’s how you can optimize your rendering setup to “rank” for AI bots and gain visibility in LLMs like ChatGPT.</p>
<h3 class="wp-block-heading">1) Audit JS-heavy content and pages</h3>
<p>The first step is to identify parts of your website that load with JavaScript and aren’t visible in the raw HTML. Use tools like <a href="https://www.oncrawl.com/" target="_blank" rel="noreferrer noopener nofollow">Oncrawl </a>and <a href="https://www.oncrawl.com/technical-seo/javascript-test-ssr-pre-rendering-implementation-oncrawl/" target="_blank" rel="noreferrer noopener nofollow">launch a crawl without JS</a>, Screaming Frog’s SEO Spider in “Text Only” crawl mode, or Chrome Developer Tools (right click > view page source) to find JS-loaded content.</p>
<p>Also, look for missing product descriptions, schema, or blog content in the page source.</p>
<h3 class="wp-block-heading">2) Choose a reliable rendering solution like Prerender.io</h3>
<p>Once you’ve identified the pages as mentioned above, it’s time to implement the right rendering solution.</p>
<p>The two main options are in-house server-side rendering or the prerendering route. While in-house SSR offers more control, it gets highly resource intensive, as mentioned earlier.</p>
<p>A prerendering solution like <a href="https://prerender.io">Prerender.io</a> allows you to integrate and go, without disrupting your site’s architecture or requiring heavy ongoing maintenance.</p>
<h3 class="wp-block-heading">3) Verify AI bots’ access</h3>
<p>The next step is ensuring AI crawlers reach your prerendered pages.</p>
<p>You can do this <a href="https://docs.prerender.io/docs/recommended-ai-crawler-user-agents-for-prerender">during the Prerender integration process</a>. After that, it’s essential to utilize tools like Oncrawl to analyze AI crawler behavior, review server logs for crawler activity, and update robots.txt to permit AI agents, such as ChatGPT-User, GPTBot, and OAI-SearchBot.</p>
<h3 class="wp-block-heading">4) Focus on prerendering high-priority pages</h3>
<p>While prerendering is cost-effective, it’s still important to make sure you aren’t rendering unnecessary pages. Focus on priority pages with important information, such as product pages, service pages, high-traffic blog posts, location pages, or support and FAQ pages. Avoid rendering any 404 pages, for example.</p>
<p>Focusing on these pages helps optimize your website’s crawl budget, ensuring that pages with the highest potential for AI visibility are prerendered and ready for AI crawlers.</p>
<h3 class="wp-block-heading">5) Optimize your content</h3>
<p>And last but not least, help AI crawlers process your content by making it easier for them. Here are some tips to make it more accessible:</p>
<ul class="wp-block-list">
<li>Ensure you have key elements present, like headings, images, schema markup, and internal links</li>
<li>Make sure your high-value content is in the initial HTML and not hidden behind tabs, lazy loading elements, or modals that may not be indexed</li>
<li>Prevent login walls or paywalls, like bot blockers or content gates, for content you want AI to index</li>
<li>Write clearly and use contextual language without jargon</li>
<li>Provide canonical tags and use consistent URLs to avoid duplicate content</li>
</ul>
<h2 class="wp-block-heading">Conclusion</h2>
<p>Optimizing for AI crawlers is no longer a choice but a must-have strategy. AI Overviews are one of the most disruptive changes in search results; <a href="https://www.semrush.com/blog/semrush-ai-overviews-study/" target="_blank" rel="noreferrer noopener nofollow">13.14% of all search results</a> triggered AI Overviews in March 2025, representing <a href="https://www.stanventures.com/news/ai-overviews-now-showing-for-54-percent-of-search-queries-2740/" target="_blank" rel="noreferrer noopener nofollow">54.6% of all search queries</a>.</p>
<p>At the pace at which AI platforms and search engines are evolving, it’s clear that they’re geared towards more adaptive, context-driven, and intelligent systems that can even understand multimedia and visual elements.</p>
<p>To optimize your website for AI visibility in the current scenario and prepare for the future, <a href="https://prerender.io/">Prerender.io</a> is an excellent solution to enhance your website’s <a href="https://prerender.io/ai-search/">AI search</a> visibility. Prerender.io ensures better AI crawling and faster indexing, supporting increased traffic, improved AI visibility, and higher conversions.</p>
<p><a href="https://auth.prerender.io/auth/realms/prerender/protocol/openid-connect/registrations?client_id=prerender-frontend&response_type=code&scope=openid%20email&redirect_uri=https://dashboard.prerender.io/integration-wizard&_gl=1*aug8l0*_gcl_au*OTc2NzA2MTk3LjE3NDU4NDgyMTc.*_ga*MjA4OTg5NTY4Ny4xNzM3NTQ4NDk3*_ga_5C99FX76HR*czE3NTIxNDY0MTckbzQ5JGcxJHQxNzUyMTQ3Mzc1JGo2MCRsMCRoMA..">Try it for free</a> to help your content reach AI search results and ChatGPT responses faster.</p>
<div class="wp-block-buttons is-content-justification-center is-layout-flex wp-container-core-buttons-is-layout-16018d1d wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link has-white-color has-text-color has-background has-link-color has-text-align-center wp-element-button" href="https://auth.prerender.io/auth/realms/prerender/protocol/openid-connect/registrations?client_id=prerender-frontend&response_type=code&scope=openid%20email&redirect_uri=https://dashboard.prerender.io/integration-wizard&_gl=1*aw60zm*_gcl_au*MTE5MTIwODkuMTc1MzgyMDU2Ng..*_ga*MjAxMjc3MTgxOC4xNzIzODAxMTYz*_ga_5C99FX76HR*czE3NTg3MTIyNTYkbzY0JGcxJHQxNzU4NzEyODYyJGo2MCRsMCRoMTU3MDU3NTEyOA.." style="border-radius:0px;background-color:#475467">Start For Free</a></div>
</div>
<h2 class="wp-block-heading">FAQs</h2>
<h3 class="wp-block-heading">What’s the difference between traditional crawlers and AI crawlers?</h3>
<p>Traditional crawlers, like Googlebot, can render JavaScript (with some limits) and index full web pages. AI crawlers, like OpenAI’s GPTBot, typically cannot execute JavaScript and only capture the raw HTML on first load. This means dynamic content often gets missed.</p>
<h3 class="wp-block-heading">Why can’t AI crawlers see my content?</h3>
<p>AI crawlers skip JavaScript execution and rely on fast, static HTML. Content hidden behind scripts (often such as product details, reviews, or tabs) may be invisible to them, leading to reduced discoverability in AI-driven search tools like ChatGPT or Perplexity.</p>
<h3 class="wp-block-heading">How does poor JavaScript rendering impact my business?</h3>
<p>If AI crawlers can’t access your product details, pricing, reviews, or blog content, your site may not appear in AI-generated answers. This can reduce visibility, harm traffic, and even create trust issues if key information is missing.</p>
<h3 class="wp-block-heading">How does prerendering improve AI visibility?</h3>
<p>Prerendering generates a fully rendered HTML version of your web pages before bots visit. AI crawlers then receive the complete content upfront (text, images, schema, and metadata), ensuring nothing is hidden.</p>
<h3 class="wp-block-heading">What pages should I prioritize for prerendering?</h3>
<p>Focus on high-value pages: product and service pages, important blog posts, FAQs, and location pages. These pages are most likely to drive visibility, conversions, and trust when surfaced in AI-powered results.</p>
<p></p>
]]></content:encoded>
</item>
<item>
<title>Google JS Rendering: Noindex No Longer Means Not Rendered</title>
<link>https://prerender.io/blog/understanding-google-noindex-rendering/</link>
<dc:creator><![CDATA[Prerender]]></dc:creator>
<pubDate>Thu, 18 Sep 2025 20:08:45 +0000</pubDate>
<category><![CDATA[Crawl Budget]]></category>
<category><![CDATA[crawl budget]]></category>
<category><![CDATA[crawl budget optimization]]></category>
<category><![CDATA[crawling]]></category>
<category><![CDATA[javascript seo]]></category>
<category><![CDATA[prerendering]]></category>
<category><![CDATA[rendering]]></category>
<guid isPermaLink="false">https://prerender.io/?p=6251</guid>
<description><![CDATA[Discover how Google's new JavaScript rendering behavior affects noindex pages and what it means for your site. Understand the implications for SEO, learn to manage crawl budgets efficiently, and explore solutions like Prerender.io to optimize content processing.]]></description>
<content:encoded><![CDATA[<p data-block-id="2kpfm">For most site owners, Google’s search pipeline feels like a black box. We know the broad strokes: Google crawls, renders, indexes, and ranks content, but the finer details are often obscure.</p>
<p data-block-id="4lqvp">That is why technical SEOs pay close attention to every shift in behavior, as even subtle changes in how Google processes websites can have big implications. One of those shifts is happening right now in JavaScript rendering: pages with noindex directives are still being rendered.</p>
<p data-block-id="5qr3c">In this article, we’ll explore Google’s new rendering behavior, why pages marked noindex are now being rendered, and what it means for your site’s health, and what SEOs should do differently.</p>
<h2 id="343qf" data-block-id="343qf">A Quick Refresher: How Google Processes Websites</h2>
<p data-block-id="37lk1">Before we dive into the main topic, let’s quickly revisit the three core stages of Google’s search pipeline:</p>
<ol type="1">
<li><strong>Crawling:</strong> this is the first stage where Googlebot discovers a page and fetches its HTML, CSS, and JS.</li>
<li><strong>Rendering:</strong> Google’s Web Rendering Service (based on Chromium) then executes the JavaScript and builds a DOM to see what the page actually displays.</li>
<li><strong>Indexing:</strong> finally, Google decides whether to store the page in its index, making it eligible to rank in search results.</li>
</ol>
<p data-block-id="gpbd">And where does <strong>noindex</strong> come in? A noindex directive (via meta tag or HTTP header) tells Google to exclude a webpage from its index. Basically, it’s like hanging a “<em>you can look, but don’t list me</em>” sign for Googlebot.</p>
<p data-block-id="245r5"><strong>Related: </strong>Learn More About <a href="https://prerender.io/blog/how-prerender-renders-javascript-websites/">How Google Crawls and Index Websites</a></p>
<h2 id="8kc45" data-block-id="8kc45">The Old Understanding: Noindex Meant No JS Rendering (and No Indexing)</h2>
<p data-block-id="2qp1r">For years, the general understanding of the SEO community was that a noindex tag would stop Google from indexing a page and prevent Googlebot from rendering its JavaScript. This is still spelled out in the <a href="https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics#:~:text=When%20Google%20encounters%20,render%20or%20index%20the%20page">Google Search Documentation</a>:</p>
<p data-block-id="18q0f">“<em>When Google encounters noindex in the robots meta tag before running JavaScript, it doesn’t render or index the page.</em>”</p>
<p data-block-id="fc7oi">For this reason, noindex became a reliable safety net that SEOs built strategies around. You could keep certain pages out of search results while still allowing Google to crawl them for link discovery. And since rendering was skipped, the assumption was that fewer resources were being used, conserving your <a href="https://prerender.io/blog/impact-of-noindex-vs-nofollow-tags/">crawl budget</a>.</p>
<p data-block-id="c4a0c">This principle of handling and managing noindex pages also shaped how technical specialists approached site architecture, navigation, and audits, particularly for <a href="https://prerender.io/blog/how-to-optimize-seo-for-large-scale-websites/">large-scale websites</a> with thousands of pages.</p>
<p data-block-id="56irh">However, recent observations suggest this long-established understanding no longer holds true.</p>
<h2 id="7pave" data-block-id="7pave">The New Understanding: Noindex No Longer Stops JS Rendering</h2>
<p data-block-id="fhru1">It appears that Google is now rendering noindex pages, at least when it comes to executing JS and handling fetch requests. To better put things into context, <strong>Dave Smart</strong>, a Technical SEO Expert, ran a series of <a href="https://tamethebots.com/blog-n-bits/noindex-does-not-mean-not-rendered">controlled tests</a> to confirm and document this behavioral shift.</p>
<p data-block-id="dlr70">He set up pages that triggered JavaScript fetch() calls to a logging endpoint and monitored Googlebot’s behavior. If Googlebot rendered the page and executed the script, the requests would appear in the server logs.</p>
<p data-block-id="an0gq">The results were conclusive across multiple test scenarios:</p>
<p data-block-id="3i4rv"><strong>Test 1: Page with <meta name=”robots” content=”noindex”> and JS fetch. </strong>Googlebot made the POST fetch request (indicating it executed the script). However, the page was still treated as noindexed and was not put in the index.</p>
<figure class="image strchf-type-image undefined strchf-size-undefined strchf-align-center" data-wp-editing="1"><picture><source srcset="https://images.storychief.com/account_57345/csnwhhwinw7-twehflxevnowodfmnnvjwecpkb5-sy67qyrpepj-hmqmlxtg-yqlulwsakeytrtfpnx6cs-zylsao4nazq_ddcf60baeba0a4a67302268175e3a92a_800.jpg 1x" media="(max-width: 768px)" /><source srcset="https://images.storychief.com/account_57345/csnwhhwinw7-twehflxevnowodfmnnvjwecpkb5-sy67qyrpepj-hmqmlxtg-yqlulwsakeytrtfpnx6cs-zylsao4nazq_ddcf60baeba0a4a67302268175e3a92a_800.jpg 1x" media="(min-width: 769px)" /><img decoding="async" src="https://images.storychief.com/account_57345/csnwhhwinw7-twehflxevnowodfmnnvjwecpkb5-sy67qyrpepj-hmqmlxtg-yqlulwsakeytrtfpnx6cs-zylsao4nazq_ddcf60baeba0a4a67302268175e3a92a_800.jpg" /></picture><figcaption></figcaption></figure>
<p data-block-id="81kiq"><strong>Test 2: Page with X-Robots-Tag: noindex (HTTP header) and JS fetch. </strong>Googlebot again performed the POST fetch. The page remained excluded from the index, consistent with Test 1.</p>
<figure class="image strchf-type-image undefined strchf-size-undefined strchf-align-center"><picture><source srcset="https://images.storychief.com/account_57345/vi0zxwy4be68dbgnqxmay0rrei-lwnwbjrfdbcuef-plhobw3vtmiisupcrnltmsmkudakeytrtfpnx6cs-zylsao4nazq_ddcf60baeba0a4a67302268175e3a92a_800.jpg 1x" media="(max-width: 768px)" /><source srcset="https://images.storychief.com/account_57345/vi0zxwy4be68dbgnqxmay0rrei-lwnwbjrfdbcuef-plhobw3vtmiisupcrnltmsmkudakeytrtfpnx6cs-zylsao4nazq_ddcf60baeba0a4a67302268175e3a92a_800.jpg 1x" media="(min-width: 769px)" /><img loading="lazy" decoding="async" class="alignnone" src="https://images.storychief.com/account_57345/vi0zxwy4be68dbgnqxmay0rrei-lwnwbjrfdbcuef-plhobw3vtmiisupcrnltmsmkudakeytrtfpnx6cs-zylsao4nazq_ddcf60baeba0a4a67302268175e3a92a_800.jpg" alt="Test 2: Page with X-Robots-Tag: noindex (HTTP header) and JS fetch" width="800" height="509" /></picture><figcaption></figcaption></figure>
<p data-block-id="7hl17"><strong>Test 3: 404 page with JS fetch. </strong>No rendering happened here. Googlebot didn’t execute the script at all, which shows that a hard 404 still stops the rendering pipeline.</p>
<figure class="image strchf-type-image undefined strchf-size-undefined strchf-align-center"><picture><source srcset="https://images.storychief.com/account_57345/ilrxxkwsvrtn7wdmaaeomtppedlncwr266b-h2cjsoorj0tyj7togaqyqkzuc-kf8umiikeytrtfpnx6cs-zylsao4nazq_e942b39c8ad4703bc0d719f3458ad2e1_800.jpg 1x" media="(max-width: 768px)" /><source srcset="https://images.storychief.com/account_57345/ilrxxkwsvrtn7wdmaaeomtppedlncwr266b-h2cjsoorj0tyj7togaqyqkzuc-kf8umiikeytrtfpnx6cs-zylsao4nazq_e942b39c8ad4703bc0d719f3458ad2e1_800.jpg 1x" media="(min-width: 769px)" /><img loading="lazy" decoding="async" class="alignnone" src="https://images.storychief.com/account_57345/ilrxxkwsvrtn7wdmaaeomtppedlncwr266b-h2cjsoorj0tyj7togaqyqkzuc-kf8umiikeytrtfpnx6cs-zylsao4nazq_e942b39c8ad4703bc0d719f3458ad2e1_800.jpg" alt="Test 3: 404 page with JS fetch" width="800" height="490" /></picture><figcaption></figcaption></figure>
<p data-block-id="fpnq4"><strong>Test 4: Noindex page with JS redirect. </strong>Googlebot executed the script and logged the POST request, but the page was still excluded and, interestingly, the redirect target wasn’t discovered or followed.</p>
<figure class="image strchf-type-image undefined strchf-size-undefined strchf-align-center"><picture><source srcset="https://images.storychief.com/account_57345/o3-mq2-ega2pvnzchcksoh6j9jx0fzounzochv2ubswwj0y7q7gv14kk8txubi7aqqingkeytrtfpnx6cs-zylsao4nazq_e942b39c8ad4703bc0d719f3458ad2e1_800.jpg 1x" media="(max-width: 768px)" /><source srcset="https://images.storychief.com/account_57345/o3-mq2-ega2pvnzchcksoh6j9jx0fzounzochv2ubswwj0y7q7gv14kk8txubi7aqqingkeytrtfpnx6cs-zylsao4nazq_e942b39c8ad4703bc0d719f3458ad2e1_800.jpg 1x" media="(min-width: 769px)" /><img loading="lazy" decoding="async" class="alignnone" src="https://images.storychief.com/account_57345/o3-mq2-ega2pvnzchcksoh6j9jx0fzounzochv2ubswwj0y7q7gv14kk8txubi7aqqingkeytrtfpnx6cs-zylsao4nazq_e942b39c8ad4703bc0d719f3458ad2e1_800.jpg" alt="Test 4: Noindex page with JS redirect" width="800" height="797" /></picture><figcaption><strong>[[Image]]</strong></figcaption></figure>
<p data-block-id="5kf6t">In short,<strong> Google still respects noindex for indexing, but it no longer skips rendering. </strong>And in Smart’s words:</p>
<p data-block-id="4m7sg">“<em>The fact that the requests made to the test API endpoint were made with a POST method, and not a GET method, gives me more confidence that these requests are being made as part of the rendering process.</em>”</p>
<p data-block-id="f3152">These findings have also been corroborated by other technical SEO experts, showing a systematic change in Google rendering behavior.</p>
<h2 id="121gg" data-block-id="121gg">Why Would Google Render Noindex Pages?</h2>
<p data-block-id="4ra89">At the surface level, this seems inefficient. Why spend resources rendering a page that will not be indexed? While Google hasn’t officially confirmed the reasoning, we can draw a straight line to these three likely drivers:</p>
<ul>
<li><strong>Detecting manipulation: </strong>some websites try to game the system by dynamically removing noindex tags via JavaScript after initial crawling. Rendering allows Google to spot these tricks.</li>
<li><strong>Extracting signals: </strong>even if a page is “noindex,” it may contain useful internal links, <a href="https://prerender.io/blog/structured-data-for-seo/" target="_blank" rel="noopener noreferrer">structured data</a>, or other signals that help Google understand the rest of the site.</li>
<li><strong>Comprehensive site analysis: </strong>rendering provides Google with a complete picture of how a website functions, including user experience metrics and technical implementation quality.</li>
</ul>
<p data-block-id="dklt9">To be fair, from Google’s perspective, this makes sense for maintaining search quality and preventing abuse. However, NOT so much for website owners as it introduces unexpected challenges.</p>
<h2 id="88adr" data-block-id="88adr">SEO Impacts of Rendering Noindex Pages</h2>
<p data-block-id="3601e">Google’s shift to rendering noindex pages affects how your site is crawled, analyzed, and reported. Here’s what you need to know:</p>
<h3 id="2os1h" data-block-id="2os1h">1. Technical Issue Visibility</h3>
<p data-block-id="apikl">Noindex tags no longer shield technical problems from Google’s analysis. Issues such as JS errors, slow load times, broken API calls, or accessibility problems previously hidden behind the noindex directive now become visible in Search Console or other <u><a href="https://prerender.io/blog/best-technical-seo-tools/">SEO testing tools</a></u>.</p>
<p data-block-id="1ka65">Now, while these pages won’t directly impact rankings, the technical issues they reveal may indicate broader <a href="https://prerender.io/resources/free-downloads/site-audit-tool/" target="_blank" rel="noopener noreferrer">site health problems</a>.</p>
<p data-block-id="u4vf"><strong>Resource:</strong> <a href="https://prerender.io/blog/technical-seo-issues/">See 19 Technical SEO Issues That Hurt Your Website Performance</a></p>
<h3 id="ei7ts" data-block-id="ei7ts">2. Signals Still Get Processed</h3>
<p data-block-id="t7hk">Noindexed pages don’t disappear from Google’s understanding of your site. Structured data, canonical tags, and internal links are still read, meaning misconfigurations can bleed into your broader site signals.</p>
<p data-block-id="4lhd5">For example, if a noindexed filter page points to the wrong canonical, it could confuse Google about which product page should be authoritative.</p>
<h3 id="4f8hl" data-block-id="4f8hl">3. Crawl Budget and Server Strain</h3>
<p data-block-id="3ogu3">It’s no secret that JavaScript rendering is resource-intensive. On small sites, this may be negligible. But on <a href="https://prerender.io/blog/ecommerce-requests-wasting-your-crawl-budget/">large ecommerce sites</a> with thousands of URLs, the extra load can waste both your server capacity and crawl budget.</p>
<p data-block-id="d140n">For example, imagine an ecommerce site with 50,000 noindex filter URLs. If each requires a few seconds of JS execution, that could mean dozens of wasted rendering hours every crawl cycle. Multiply that across thousands of fetches, and you’re looking at serious server strain and crawl budget drain that could have been allocated to indexable revenue-driving pages.</p>
<h3 id="5ichn" data-block-id="5ichn">4. Reporting Complexity</h3>
<p data-block-id="8q8vl">Because Google renders these pages, diagnostic tools may flag irrelevant issues, creating noise in your SEO reporting. As a result, you’ll now need to filter out this noise and also diagnose rendering issues that genuinely impact site health.</p>
<h2 id="ch7uk" data-block-id="ch7uk">How Can Marketers and SEOs Handle This New Behavior?</h2>
<p data-block-id="bjfgt">The impacts we’ve covered clearly point to this: marketers and SEO specialists can no longer treat noindex as a simple “end of the line.” Instead, they must be deliberate in optimizing what Google sees and processes, especially when resources and crawl budget management matter<strong>.</strong></p>
<h3 id="fmd8f" data-block-id="fmd8f">Apply the Proper Directive Strategy</h3>
<p data-block-id="erk2n">Whether it’s a noindex or robots.txt tag, a clean directive strategy helps prevent wasted crawl budget and also helps Google allocate resources more efficiently across your site. However, too often, these two get lumped together as if they solve the same problem, but they do not.</p>
<ul>
<li><strong>Noindex: </strong>best for pages that can be crawled and rendered, but should not appear in Google’s search results (e.g., thank-you pages or private members-only content).</li>
<li><strong>Disallow (robots.txt): </strong>best for pages that you want Google to completely ignore crawling (e.g., admin areas, private user data, or staging environments). Google will not fetch or render these pages.</li>
</ul>
<p data-block-id="bpdce"><strong>Note: </strong>never use both noindex and disallow tags on the same URL, as this prevents Google from seeing the noindex instruction. Check out our guide for detailed instructions on <a href="https://prerender.io/blog/robots-txt-and-seo/">how to apply robots.txt directive to your website</a>.</p>
<h3 id="b2okm" data-block-id="b2okm">Expand Your Technical Audit Requirements</h3>
<p data-block-id="av1gs"><a href="https://prerender.io/blog/how-to-conduct-a-technical-seo-audit/">SEO audits</a> that only check whether a page is “indexed or not” are no longer enough. Audits now need to:</p>
<ul>
<li><strong>Analyze server logs or GSC Crawl Stats </strong>to identify how much Googlebot activity is spent on noindex pages.</li>
<li><strong>Compare raw HTML vs. rendered HTML </strong>to identify signals or errors hiding behind JS execution. Tools like <a href="https://prerender.io/prerender-vs-screaming-frog/" target="_blank" rel="noopener noreferrer">Screaming Frog</a>, Sitebulb, or Google’s URL Inspection tool can help you detect mismatches.</li>
<li><strong>Check for conflicting directives</strong>, like pages marked noindex but also blocked in robots.txt, or canonical tags pointing to noindexed pages.</li>
<li><strong>Evaluate internal linking and sitemaps </strong>to ensure noindex pages are not heavily linked from your main navigation or included in XML sitemaps.</li>
</ul>
<h3 id="eb28m" data-block-id="eb28m">Implement a JS Prerendering Solution</h3>
<p data-block-id="5p247">Several approaches, such as <a href="https://prerender.io/blog/what-is-srr-and-why-do-you-need-to-know/" target="_blank" rel="noopener noreferrer">self-built SSR</a>, static rendering, and hydration, can improve how search engines process JavaScript, but they’re costly to build and maintain. Instead, a prerendering solution like <strong>Prerender.io</strong> offers a more powerful and efficient alternative for saving crawl budget.</p>
<p data-block-id="4lqdo"><a href="http://prerender.io/">Prerender.io</a> is a dynamic JS rendering solution that serves fast, static HTML versions of your JS-heavy pages directly to search engine crawlers such as Googlebot and Bingbot. This improves your crawl efficiency, reduces server load, and avoids the high costs of maintaining an in-house rendering setup.</p>
<p data-block-id="aomcj">It also helps ensure consistent handling of noindex directives, so your pages are rendered and processed exactly as intended. In other words, as Google’s rendering behavior evolves, Prerender.io gives you stability and technical control.</p>
<p data-block-id="62uc3"><a href="https://prerender.io/blog/a-guide-to-prerender-process-and-benefits/">Learn how Prerender.io works</a> in more detail and <a href="https://prerender.io/comparison/">how Prerender.io compares to other solutions</a>.</p>
<figure class="image strchf-type-image undefined strchf-size-undefined strchf-align-center"><picture><source srcset="https://images.storychief.com/account_57345/iqnqvyvkzmcxebjava9oesncese4ylxohhgfn2giuhz1dihgr3xatrxzrlagltskbvavokeytrtfpnx6cs-zylsao4nazq_e942b39c8ad4703bc0d719f3458ad2e1_800.jpg 1x" media="(max-width: 768px)" /><source srcset="https://images.storychief.com/account_57345/iqnqvyvkzmcxebjava9oesncese4ylxohhgfn2giuhz1dihgr3xatrxzrlagltskbvavokeytrtfpnx6cs-zylsao4nazq_e942b39c8ad4703bc0d719f3458ad2e1_800.jpg 1x" media="(min-width: 769px)" /><img loading="lazy" decoding="async" class="alignnone" src="https://images.storychief.com/account_57345/iqnqvyvkzmcxebjava9oesncese4ylxohhgfn2giuhz1dihgr3xatrxzrlagltskbvavokeytrtfpnx6cs-zylsao4nazq_e942b39c8ad4703bc0d719f3458ad2e1_800.jpg" alt="Prerender vs. SSR vs. Static Rendering vs. Hydration" width="800" height="563" /></picture><figcaption></figcaption></figure>
<h2 id="476qt" data-block-id="476qt">Improve Website JavaScript Rendering With Prerender.io</h2>
<p data-block-id="blfnj">Again, while Google itself hasn’t officially confirmed this new rendering pattern (and it’s possible the behavior could evolve again), evidence from multiple SEO experts has pointed to it, and site owners can’t afford to ignore it.</p>
<p data-block-id="bvsfk">The takeaway is simple: <strong>refine your directive strategy, broaden your audit scope, and make sure Google’s resources are spent on the pages that drive real value through effective dynamic content indexing.</strong></p>
<p data-block-id="3pvlr">Also, by integrating <strong>Prerender.io</strong>, you can easily solve your site’s technical SEO issues, bid crawl budget troubles goodbye, and future-proof your SEO strategy. It is <a href="https://prerender.io/blog/how-to-install-prerender/">easy to install</a> and use, and fits right into your <a href="https://prerender.io/framework/">JS-based framework</a>, so there’s no need to change any of your tech stack.</p>
<p data-block-id="88g32">Enjoy the rendering benefits for your JavaScript website <a href="https://prerender.io/pricing/">by adopting Prerender.io for free today</a>!<!-- End strchf script --></p>
]]></content:encoded>
</item>
<item>
<title>Screaming Frog Auditing Blindspot for JavaScript Rendering</title>
<link>https://prerender.io/blog/screaming-frog-auditing-for-javascript-rendering/</link>
<dc:creator><![CDATA[Prerender]]></dc:creator>
<pubDate>Tue, 09 Sep 2025 07:48:17 +0000</pubDate>
<category><![CDATA[Javascript SEO]]></category>
<category><![CDATA[javascript seo]]></category>
<guid isPermaLink="false">https://prerender.io/?p=6231</guid>
<description><![CDATA[Uncover Screaming Frog’s auditing limitations in JavaScript rendering and how they can affect your SEO audit.]]></description>
<content:encoded><![CDATA[<figure class="strchf-table"></figure>
<p data-block-id="4qp8f">Many SEO teams use Screaming Frog’s SEO spider to audit JavaScript rendering issues. But here’s one Screaming Frog limitation you may not know: its crawlers don’t perfectly mirror Googlebot’s behaviour.</p>
<p data-block-id="6444h">Screaming Frog can miss timeout-dependent content, overlook dynamically generated URLs, and fail to discover elements that load asynchronously, creating dangerous blind spots in your SEO analysis.</p>
<p data-block-id="3qc12">Discover some Screaming Frog JS rendering blindspots, understand their impact on your JavaScript SEO strategy, and, most importantly, show you how to fix these hidden crawlability issues before they tank your search visibility. We’ll also show you how Prerender.io is a complementary technical SEO tool that plugs these JavaScript rendering gaps entirely.</p>
<h2 id="dj044" data-block-id="dj044">How Screaming Frog’s Rendering Differs From Googlebot (An SEO Audit Blindspot)</h2>
<p data-block-id="6db40">Search engine crawlers like Googlebot can read JavaScript-generated content, but they don’t behave like real people. They don’t click buttons, scroll down pages, or trigger events the way users do. For instance, when vital content or links only show up after a user clicks a “Load More” button, crawlers may never see them.</p>
<p data-block-id="hgk8">There’s also a timing problem. Crawlers only wait so long for a page to load before they move on. If your JavaScript is slow or takes extra time to finish running, the crawler snaps a picture of the page before everything has loaded. Anything that loads late may be left out and never get indexed.</p>
<p data-block-id="274iq">Now, the catch is that crawlers don’t all handle this waiting period the same way. Googlebot is more patient and can prioritize or retry resources to capture a fuller page. Screaming Frog, on the other hand, stops earlier, which means it doesn’t always replicate how Googlebot renders JavaScript-heavy content.</p>
<p data-block-id="4icas">This mismatch creates a disconnect—a blindspot in Screaming Frog audits: <strong>a page might look fine in Screaming Frog but still fail to index properly in Google.</strong></p>
<h2 id="6qtvg" data-block-id="6qtvg">What Are The Screaming Frog JavaScript Rendering Limitations?</h2>
<p data-block-id="9ss1o">Screaming Frog is an excellent SEO auditing tool. Their SEO Spider’s JavaScript rendering solution uses a headless Chromium browser (similar to Googlebot) to crawl dynamic pages effectively.</p>
<p data-block-id="9ss1o"><img loading="lazy" decoding="async" class="wp-image-6232 size-full aligncenter" src="https://prerender.io/wp-content/uploads/3vtkvsp0x4ikryh9wlei3zlwrg7g9s1mi-9sd1fkykx8pjqpi52-ouhufqjq4fglpocmykey3abkglpnukolwiylfqouqa_7f65f51391ebb52fabb1dffb3ac6641f_2000.webp" alt="Screaming Frog spider configuration" width="850" height="126" srcset="https://prerender.io/wp-content/uploads/3vtkvsp0x4ikryh9wlei3zlwrg7g9s1mi-9sd1fkykx8pjqpi52-ouhufqjq4fglpocmykey3abkglpnukolwiylfqouqa_7f65f51391ebb52fabb1dffb3ac6641f_2000.webp 850w, https://prerender.io/wp-content/uploads/3vtkvsp0x4ikryh9wlei3zlwrg7g9s1mi-9sd1fkykx8pjqpi52-ouhufqjq4fglpocmykey3abkglpnukolwiylfqouqa_7f65f51391ebb52fabb1dffb3ac6641f_2000-300x44.webp 300w, https://prerender.io/wp-content/uploads/3vtkvsp0x4ikryh9wlei3zlwrg7g9s1mi-9sd1fkykx8pjqpi52-ouhufqjq4fglpocmykey3abkglpnukolwiylfqouqa_7f65f51391ebb52fabb1dffb3ac6641f_2000-768x114.webp 768w" sizes="(max-width: 850px) 100vw, 850px" /></p>
<p data-block-id="d5qnl">In theory, this lets Screaming Frog fetch the initial HTML, execute JavaScript, and capture the post-render version of the page just like a browser would. In practice, however, there are several Screaming Frog limitations that create web SEO audit blind spots.</p>
<h3 id="8p7nd" data-block-id="8p7nd">1. Screaming Frog Sets A Snapshot Time (Ajax Timeout)</h3>
<p data-block-id="52rem">By default, Screaming Frog waits exactly 5 seconds for your JavaScript page to execute and for the content to load before taking its HTML snapshot. For many sites, this works fine. For others, it’s a recipe for missing content.</p>
<p data-block-id="7khmh"><img loading="lazy" decoding="async" class="aligncenter wp-image-6233 size-full" src="https://prerender.io/wp-content/uploads/6a1sfft7rezjkbtqk3w1winalbhdjinci5w8aeke-fzgaul15c3wdxdvwexeqtlyi-hvakey3abkglpnukolwiylfqouqa_7f65f51391ebb52fabb1dffb3ac6641f_2000.webp" alt="Screaming Frog's JS rendering limit" width="535" height="282" srcset="https://prerender.io/wp-content/uploads/6a1sfft7rezjkbtqk3w1winalbhdjinci5w8aeke-fzgaul15c3wdxdvwexeqtlyi-hvakey3abkglpnukolwiylfqouqa_7f65f51391ebb52fabb1dffb3ac6641f_2000.webp 535w, https://prerender.io/wp-content/uploads/6a1sfft7rezjkbtqk3w1winalbhdjinci5w8aeke-fzgaul15c3wdxdvwexeqtlyi-hvakey3abkglpnukolwiylfqouqa_7f65f51391ebb52fabb1dffb3ac6641f_2000-300x158.webp 300w" sizes="(max-width: 535px) 100vw, 535px" /></p>
<p data-block-id="7khmh">If a page is too slow to load content (maybe it’s waiting for API responses or large JavaScript bundles), some elements might not render within that window frame. When this happens, Screaming Frog may miss crawling it entirely and won’t mention it in the auditing report. By default, Screaming Frog waits exactly 5 seconds for your JavaScript page to execute and for the content to load before taking its HTML snapshot. For many sites, this works fine. For others, it’s a recipe for missing content.</p>
<h3 id="j69f" data-block-id="j69f">2. Screaming Frog Doesn’t Interact with Content</h3>
<p data-block-id="ajtsp">It’s important to know that Screaming Frog doesn’t click, hover, or scroll content. It loads your page, waits, and that’s it. This means that any content that requires user interaction stays invisible to its crawler. Some examples are:</p>
<ul>
<li>“Load More” buttons that reveal additional content</li>
<li>Scroll-triggered content loading</li>
<li>Click-to-reveal sections, e.g., “show more” on ecommerce product descriptions</li>
<li>Hover-activated elements, e.g., zooming in on product images</li>
</ul>
<p data-block-id="2v7fd">This JavaScript user-interactive content loads seamlessly for users browsing your site, but it doesn’t exist for search engine crawlers. Unless you have fallback links or server-rendered alternatives, these sections represent massive holes in your site’s crawlability and SEO performance.</p>
<h3 id="430em" data-block-id="430em">3. Resource Loading Differences Cause Systematic Blindspots</h3>
<p data-block-id="8gi50">Screaming Frog requires full access to JavaScript, CSS, and image resources to render pages accurately. In practice, certain resources often fail to load because they’re blocked or behave differently for non-browser agents. Common issues include:</p>
<ul>
<li>JavaScript files blocked by robots.txt. See these <u><a href="https://prerender.io/blog/robots-txt-and-seo/">robots.txt best practices</a></u> to mitigate this issue.</li>
<li>API endpoints returning errors for non-browser agents</li>
<li>CDN timeouts during resource loading</li>
<li>Third-party scripts that fail silently</li>
</ul>
<p data-block-id="fp4lp">Why it matters: if critical JavaScript-driven resources are inaccessible, page rendering can break entirely. Googlebot, however, handles these scenarios with more advanced resource prioritization and fallback mechanisms. This creates a gap between what Screaming Frog reports and what Google actually indexes.</p>
<p data-block-id="9m6je">As a result, Screaming Frog may show a page as fully rendered, while search engines fail to index key content due to timeouts, blocked files, or elements that only appear after user interaction.</p>
<h2 id="arodo" data-block-id="arodo">Two Case Studies on the Impact of JavaScript Rendering Failures</h2>
<p data-block-id="tash">The financial and traffic impact of JavaScript rendering can be huge, with some organizations losing millions in organic search value before identifying the root causes.</p>
<p data-block-id="1oqcc">Hulu, for example, suffered a<a href="https://smart.linkresearchtools.com/linkthing/case-studies/hulu-javascript-fail-visibility-drop" target="_blank" rel="noopener"> <u>56% search visibility</u></a> drop when its JavaScript implementation began serving 404 errors to Googlebot while showing normal content to users. Their homepage was deindexed entirely from Google, and tens of thousands of TV show and movie pages returned 404 errors to crawlers while maintaining 200 status codes for regular browsers. This issue persisted for over three weeks, representing millions in lost search traffic value.</p>
<p data-block-id="2tcee">Another large React-based publisher experienced a <u><a href="https://momenticmarketing.com/case-studies/javascript-seo" target="_blank" rel="noopener">51% organic traffic loss</a></u> from seemingly minor implementation issues that standard JavaScript rendering audits and Screaming Frog crawls missed. Their primary problems were:</p>
<ul>
<li>Navigation elements styled as links but not implemented with <em><a href></em> tags</li>
<li>Missing canonical tags on all URLs despite 200 status responses</li>
<li>Absent H1 tags across the entire site</li>
</ul>
<p data-block-id="acj42">These case studies highlight a recurring challenge in JavaScript SEO: sites appear functionally perfect in development and browser testing but fail systematically when search engines attempt to crawl and index the content.</p>
<p data-block-id="560s0">When critical content isn’t crawlable or visible to search engines, it doesn’t exist for SEO. Google and AI-powered search platforms won’t index or rank it, leaving your content invisible despite passing Screaming Frog audits.</p>
<p data-block-id="36jf9">But enough about the problem, let’s talk about how you can fix Screaming Frog limitations in JS rendering by using Prerender.io.</p>
<h2 id="2s7bo" data-block-id="2s7bo">How Prerender.io Fills In Screaming Frog’s Limitations</h2>
<p data-block-id="9obsb">While Screaming Frog’s JavaScript rendering solution is powerful for technical SEO auditing, its limitations with JavaScript-heavy sites mean you need a second pair of eyes to ensure your content is accessible for search engine bots and AI crawlers. For this, we recommend <u><a href="http://prerender.io/">Prerender.io</a></u>, a JavaScript content prerendering solution.</p>
<p data-block-id="bp8rp">How Prerender.io works:</p>
<ul>
<li>Uses headless Chromium (via Puppeteer) to fully render each page server-side.</li>
<li>Generates a cached static HTML snapshot.</li>
<li>Delivers this pre-rendered HTML instantly to search engine bots and AI crawlers.</li>
</ul>
<p data-block-id="6tnqf">In simple terms, Prerender.io executes all JavaScript before serving the page, ensuring search engine crawlers and bots get 100% of your content. The result is a fast, accurate, ready-to-index page that Google, Bing, and Google AI Overview can crawl without having to render the content themselves. See <a href="https://prerender.io/prerender-vs-screaming-frog/">the differences and similarities between Prerender.io and Screaming Frog.</a></p>
<p data-block-id="6ii6l">What Prerender.io surfaces that Screaming Frog may miss:</p>
<ul>
<li><strong>Dynamic content</strong>, such as product descriptions, reviews, and structured data added via JavaScript.</li>
<li><strong>JavaScript-generated navigation</strong>, such as links or routes hidden behind client-side events.</li>
<li><strong>Lazy-loaded or hidden elements</strong>, such as product listings, tabs, or meta tags, are injected after page load.</li>
</ul>
<p data-block-id="2hlq2">In short, Prerender.io reveals the full picture and surfaces hidden tabs that crawlers like Screaming Frog SEO spiders might overlook. The best part is, you can<u><a href="https://prerender.io/pricing/"> get started with Prerender.io for free</a></u> right now to start enjoying these benefits.</p>
<h2 id="2l8bc" data-block-id="2l8bc">Combining Screaming Frog and Prerender.io for Effective JavaScript Rendering</h2>
<p data-block-id="20i62">You don’t have to choose between either Prerender.io or Screaming Frog, as they both work perfectly fine together to create a comprehensive audit workflow.</p>
<p data-block-id="3unfi"><strong>Prerender.io</strong> fixes the visibility problem at the root by making JavaScript content crawlable, while <strong>Screaming Frog</strong> verifies and supplements the crawling and indexing performance. The result: a streamlined SEO and AI SEO monitoring process that ensures no content is hidden from or missed by search engines.</p>
<p data-block-id="8f0uv">Here’s how Prerender.io and Screaming Frog can work together:</p>
<ol type="1">
<li><strong>Deploy Prerender.io</strong> in production to serve static HTML snapshots to bots</li>
<li><strong>Run Screaming Frog with JavaScript rendering</strong> to validate those pages</li>
<li><strong>Compare raw vs. rendered HTML</strong> to confirm titles, headings, links, and structured data are present</li>
<li><strong>Leverage Screaming Frog’s auditing features</strong> to catch any remaining technical SEO or crawlability issues</li>
</ol>
<p data-block-id="9jaoc">In essence, Prerender.io eliminates rendering SEO gaps, and Screaming Frog confirms the solution while monitoring overall site health.</p>
<p data-block-id="2a1ju">By pairing the two SEO tools, you:</p>
<ul>
<li>Guarantee full visibility of your content to crawlers</li>
<li>Supplement crawl data with a full JS-rendered view</li>
<li>Troubleshoot rendering and indexing issues faster</li>
</ul>
<p data-block-id="1dg8">Together, Prerender.io and Screaming Frog give you a reliable, scalable way to audit and optimize even the most JavaScript-heavy websites for search visibility.</p>
<p data-block-id="c0g26">Learn more about <a href="https://prerender.io/blog/screaming-frog-vs-prerender-for-javascript-seo-auditing/">when to use Screaming Frog and Prerender.io in this blog.</a></p>
<h2 id="agplp" data-block-id="agplp">Build a Powerful Rendering SEO Workflow with Prerender.io and Screaming Frog</h2>
<p data-block-id="3u1ig">JavaScript doesn’t have to be a barrier to search visibility. Screaming Frog is excellent for finding where crawlers struggle, but it doesn’t fix the problem. That’s where Prerender.io comes in. By serving fully rendered pages to bots, Prerender.io removes the rendering vs. indexing blind spots that cost you SEO rankings and sales traffic.</p>
<p data-block-id="8bhqi">So stop losing traffic to invisible content now. <u><a href="https://prerender.io/pricing/">Get started with Prerender.io</a></u> today for free and see exactly what search engines have been missing on your JavaScript site.</p>
<p><!-- strchf script --><script>if(window.strchfSettings === undefined) window.strchfSettings = {};window.strchfSettings.stats = {url: "https://prerender.storychief.io/en/screaming-frog-javascript-rendering-68Pj6gWPwkkM9B?id=1428135578&type=2",title: "Screaming Frog Auditing Blindspot for JavaScript Rendering",siteId: "20338",id: ""};(function(d, s, id) {var js, sjs = d.getElementsByTagName(s)[0];if (d.getElementById(id)) {window.strchf.update(); return;}js = d.createElement(s); js.id = id;js.src = "https://d37oebn0w9ir6a.cloudfront.net/scripts/v0/strchf.js";js.async = true;sjs.parentNode.insertBefore(js, sjs);}(document, 'script', 'storychief-jssdk'))</script><!-- End strchf script --></p>
<p></p>
]]></content:encoded>
</item>
<item>
<title>Zero Click Searches in Ecommerce: How to Adapt to the New AI Shopping Experience</title>
<link>https://prerender.io/blog/ai-shopping-experience/</link>
<dc:creator><![CDATA[Prerender]]></dc:creator>
<pubDate>Thu, 14 Aug 2025 13:55:22 +0000</pubDate>
<category><![CDATA[Ecommerce SEO]]></category>
<category><![CDATA[ecommerce]]></category>
<category><![CDATA[ecommerce AI SEO]]></category>
<category><![CDATA[ecommerce products]]></category>
<guid isPermaLink="false">https://prerender.io/?p=6101</guid>
<description><![CDATA[Shoppers decide without visiting sites. Learn how AI shopping and zero-click searches work and how to keep your products discoverable.]]></description>
<content:encoded><![CDATA[<p data-block-id="1lis8">Online shopping is no longer confined to ecommerce stores and search engine results. Platforms like ChatGPT and Perplexity are becoming personal AI shopping assistants, answering questions, comparing products, and even facilitating purchases <em>without your customers ever visiting a website</em>.</p>
<p data-block-id="dp6es">This shift is already visible in the numbers: according to a SparkToro study, over <u><a href="https://sparktoro.com/blog/2024-zero-click-search-study-for-every-1000-us-google-searches-only-374-clicks-go-to-the-open-web-in-the-eu-its-360/https://sparktoro.com/blog/2024-zero-click-search-study-for-every-1000-us-google-searches-only-374-clicks-go-to-the-open-web-in-the-eu-its-360/">60%</a></u> of Google searches now end without a single click. More shoppers are skipping websites entirely, finding what they need directly inside these AI experiences.</p>
<p data-block-id="c4jpd">Traditional traffic is shrinking, which means visibility is the new currency. If your products aren’t discoverable on AI-driven platforms, you don’t exist.</p>
<p data-block-id="8lj5v">In this article, we’ll break down what zero-click searches are, how AI shopping platforms are reshaping consumer behavior, what it means for ecommerce, and how you can adapt.</p>
<h2 id="at8tf" data-block-id="at8tf">What is Zero Click Search?</h2>
<p data-block-id="b3mah">A zero-click search happens when a user receives the answer they’re looking for directly on the search engine results page (SERP) or AI interface, without needing to click through to a website. Historically, this came in the form of:</p>
<ul>
<li>Featured snippets and People Also Ask boxes.</li>
<li>Knowledge panels and local map packs.</li>
<li>Shopping carousels with prices and reviews.</li>
</ul>
<p data-block-id="104jo">Now, zero-click has expanded to:</p>
<ul>
<li>AI-generated summaries (Google AI Overviews and Bing Copilot answers).</li>
<li>AI Chat interfaces like ChatGPT and Claude.</li>
<li>Voice and virtual assistants (Alexa, Siri, Google Assistant).</li>
</ul>
<p data-block-id="bikjb">For example, here’s what it will look like if you search for “<em>What is <u><a href="https://prerender.io/blog/content-seo-challenges-on-magento/">Magento</a></u> commerce?</em>” You’ll get an AI-generated summary at the top of the results page—no click or website visit needed.</p>
<figure class="image strchf-type-image regular strchf-size-regular strchf-align-center"><picture><source srcset="https://images.storychief.com/account_57345/zero-click-search-example_039afd45134e90e8a880048282687a0b_800.jpg 1x" media="(max-width: 768px)" /><source srcset="https://images.storychief.com/account_57345/zero-click-search-example_039afd45134e90e8a880048282687a0b_800.jpg 1x" media="(min-width: 769px)" /><img decoding="async" src="https://images.storychief.com/account_57345/zero-click-search-example_039afd45134e90e8a880048282687a0b_800.jpg" alt="Google AI Overview example." /></picture></figure>
<h2 id="ce315" data-block-id="ce315">How Does Zero Click Search Work?</h2>
<p data-block-id="91hem">When someone types a query, Google or Bing first pulls from their indexed pages. Sites that use structured data often rank better in these results because the markup helps search engines identify and display the necessary information in snippets, knowledge panels, or other features.</p>
<p data-block-id="d94jb">AI has pushed this further by powering AI Overviews and large language models (LLMs) that synthesize data into conversational summaries. It also drives conversational commerce, where AI chat and search tools move users from questions to product recommendations without traditional click-throughs.</p>
<p data-block-id="2ll5h">However, whether a zero-click result shows depends on how the search engine reads user intent—signals like phrasing, search history, and behavior help determine if someone wants quick facts, comparisons, or is ready to buy. If the query can be satisfied directly, the engine delivers it.</p>
<p data-block-id="73pst">Resource: see the comparison between <u><a href="https://prerender.io/blog/traditional-search-vs-ai-powered-search-explained/">traditional search vs. AI-powered search</a></u>.</p>
<h2 id="108gb" data-block-id="108gb">Why are Zero-Click Searches Becoming More Prominent?</h2>
<p data-block-id="4ksmk">First, today’s shoppers, especially among the <u><a href="https://www.statista.com/topics/11087/gen-z-online-shopping-behavior/#topicOverview">younger demographics</a></u>, have an appetite for instant, personalized answers. Rather than clicking through multiple website links or sifting through cluttered ecommerce pages, they’re turning to AI platforms to get quick information upfront.</p>
<p data-block-id="fm7gv">Like <u><a href="https://www.linkedin.com/in/lindseymazza?utm_source=share&utm_campaign=share_via&utm_content=profile&utm_medium=ios_app">Lindsey Mazza</a></u>, Global Retail Lead at <u><a href="https://www.capgemini.com/news/press-releases/71-of-consumers-want-generative-ai-integrated-into-their-shopping-experiences/">Capgemini</a></u>, says: <em>“Consumers today want personalized shopping experiences, enhanced by AI and generative AI. In addition, they expect fast and efficient deliveries and have become more conscious of their purchasing impact.”</em></p>
<p data-block-id="288a">Mobile browsing has also amplified this trend, with over <u><a href="https://scrumdigital.com/blog/zero-click-search-trends-google-serp-analysis/#:~:text=That%E2%80%99s%20a%20significant%20leap%20from,major%20shift%20in%20user%20behavior">75% of mobile searches</a></u> last year ending without a site visit. Users, in response to the friction that comes with navigating sites on a small screen, are turning to zero-click formats that give them what they need immediately (summaries, recommendations, or product suggestions).</p>
<p data-block-id="9541o">At the same time, search engines and AI platforms benefit from keeping users around. The more they answer queries directly, the longer users stay on their platforms, giving them valuable behavior data, increased engagement, and ad opportunities</p>
<p data-block-id="9iet8">Finally, zero-click results help with decision fatigue. With countless options and conflicting reviews online, users prefer AI-curated recommendations to simplify choices and save time.</p>
<p data-block-id="e03gb">In fact, as of July 2025, ChatGPT has nearly <u><a href="https://www.demandsage.com/chatgpt-statistics/">800 million</a></u> weekly active users, while Perplexity also boasts of <u><a href="https://www.demandsage.com/perplexity-ai-statistics/">2 million</a></u> daily users. These are just two of many tools to show how LLMs are fast becoming the default starting points for millions of users, bypassing the traditional Google-to-website path entirely.</p>
<p data-block-id="231tt">Resource: <u><a href="https://prerender.io/blog/seo-vs-aio-vs-geo/">see how search behavior differs across SEO, AIO, and GEO</a></u>.</p>
<h2 id="fo891" data-block-id="fo891">The New AI Shopping Experience: Chatbots, Shopping Overviews, and LLMs</h2>
<p data-block-id="tpv9">AI is becoming the main character in ecommerce. Recent surveys show that <u><a href="https://blog.adobe.com/en/publish/2025/03/17/adobe-analytics-traffic-to-us-retail-websites-from-generative-ai-sources-jumps-1200-percent">47% of users</a></u> rely on generative AI for product recommendations, while 55% use it to research products before buying.</p>
<p data-block-id="1nnp7">Voice assistants are part of the funnel too, with nearly a <u><a href="https://learn.g2.com/voice-search-statistics">quarter of consumers</a></u> purchasing through tools like Alexa or Siri, and 40% using them to verify product details.</p>
<p data-block-id="cfr6s">For brands, this signals a fundamental shift as discovery, comparison, and even purchase decisions are increasingly happening off traditional websites and on AI-powered platforms.</p>
<p data-block-id="d54to">Google is currently leading this new AI shopping experience. In October 2024, it overhauled its <u><a href="https://blog.google/products/shopping/google-shopping-ai-update-october-2024/">Shopping experience</a></u> by merging its 45-billion-product Shopping Graph with the Gemini AI model. A single query now produces AI-generated summaries of what to consider, top product picks, and category filters.</p>
<figure class="image strchf-type-image regular strchf-size-regular strchf-align-center"><picture><source srcset="https://images.storychief.com/account_57345/google-ai-shopping-example_66cc9d6a0cb8c4e918f5b5f9bdf7b1df_800.jpg 1x" media="(max-width: 768px)" /><source srcset="https://images.storychief.com/account_57345/google-ai-shopping-example_66cc9d6a0cb8c4e918f5b5f9bdf7b1df_800.jpg 1x" media="(min-width: 769px)" /><img decoding="async" src="https://images.storychief.com/account_57345/google-ai-shopping-example_66cc9d6a0cb8c4e918f5b5f9bdf7b1df_800.jpg" alt="Google AI shopping example." /></picture></figure>
<p data-block-id="525li">In many cases, shoppers never even reach a retailer’s website—Google’s Shopping tab offers personalized feeds, deal finders, and even virtual try-ons, turning the search engine into a full-fledged AI shopping assistant.</p>
<p data-block-id="796nl">OpenAI has also made similar moves with ChatGPT. In April 2025, it introduced <u><a href="https://openai.com/chatgpt/search-product-discovery/">native shopping capabilities</a></u> that let users research, compare, and buy products through the chat interface.</p>
<p data-block-id="3ev58">During a prelaunch demo, <u><a href="https://www.linkedin.com/in/adam-h-fry?utm_source=share&utm_campaign=share_via&utm_content=profile&utm_medium=ios_app">Adam Fry</a></u>, Search Product Lead, showed how the tool can surface tailored recommendations complete with pricing, availability, attributes, and direct purchase links.</p>
<figure class="image strchf-type-image regular strchf-size-regular strchf-align-center"><picture><source srcset="https://images.storychief.com/account_57345/open-ai-shopping-example_93ebbdae1a83c29e303c758d1f0bb14b_800.jpeg 1x" media="(max-width: 768px)" /><source srcset="https://images.storychief.com/account_57345/open-ai-shopping-example_93ebbdae1a83c29e303c758d1f0bb14b_800.jpeg 1x" media="(min-width: 769px)" /><img loading="lazy" decoding="async" class="alignnone" src="https://images.storychief.com/account_57345/open-ai-shopping-example_93ebbdae1a83c29e303c758d1f0bb14b_800.jpeg" alt="Open AI shopping example." width="800" height="467" /></picture></figure>
<p data-block-id="1im7g">Other AI recommendation engines, including Perplexity and Bing Copilot, are chasing the same goal. Perplexity Shopping, for instance, now offers users unbiased recommendations, visual product discovery, and seamless checkout contained within its platform.</p>
<p data-block-id="c33qj">These innovations all point to the reality that AI is reshaping the customer’s shopping journey. And when AI sits between brands and their buyers, the big question is how this shift will impact <u><a href="https://prerender.io/blog/sudden-drop-in-traffic/">traffic</a></u>, conversions, and overall bottom line.</p>
<h2 id="q7uv" data-block-id="q7uv">What are the Implications of Zero-Click Searches on Traffic and Conversions?</h2>
<p data-block-id="56uvj">So, what happens when most of your potential customers never make it past the search results? Let’s look at how zero-click searches create challenges for ecommerce brands.</p>
<h3 id="3hjue" data-block-id="3hjue">1. Declining Organic Traffic</h3>
<p data-block-id="km3c">Page-one rankings no longer guarantee brand visibility. AI Overviews, featured snippets, and knowledge panels dominate above-the-fold space, often burying organic listings. So, even if you rank, CTRs are low, which means fewer opportunities to engage and convert users.</p>
<h3 id="49qke" data-block-id="49qke">2. Loss of Control Over Messaging</h3>
<p data-block-id="5nfis">AI often rewrites or condenses your content, meaning your brand could be mentioned without the context you’d prefer—or skipped entirely if your content isn’t structured for these systems. That can dilute your messaging or, worse, make your competitors more prominent.</p>
<h3 id="5v954" data-block-id="5v954">3. Limited Data and Targeting Opportunities</h3>
<p data-block-id="64t52">With fewer site visits, your first-party data shrinks. This means fewer cookies, less behavioral insight, and limited ability to retarget users. Combined with murky attribution—where buyers discover you in AI results but convert elsewhere—it becomes harder to track ROI accurately.</p>
<h3 id="3it02" data-block-id="3it02">4. Competitive Displacement</h3>
<p data-block-id="cm7ct">AI-driven answers often prioritize content that aligns with its training data, structure, or intent signals. If your pages don’t meet those signals, your competitors can edge you out, even if you previously outranked them in traditional SERPs.</p>
<h2 id="b1bek" data-block-id="b1bek">Should Ecommerce Stores be Worried about Zero-Click Search and AI Shopping?</h2>
<p data-block-id="4vpvo">Not necessarily. Zero-click search is disruptive, but it doesn’t automatically spell doom for ecommerce brands. The impact also depends on how businesses choose to respond.</p>
<p data-block-id="dfm88">Yes, fewer people may be clicking through to your site, but this type of search can still work in your favor if you position your brand strategically.</p>
<p data-block-id="em58i">Being consistently featured in AI results can build visibility, top-of-mind awareness, drive trust, and influence purchase decisions—whether on your site or through AI shopping assistants.</p>
<h2 id="ft0ak" data-block-id="ft0ak">How to Optimize Your Ecommerce Website for Zero-Click and AI Shopping</h2>
<p data-block-id="9mqgd">To remain discoverable and measurable in an ecosystem where clicks are no longer the primary signal of success, businesses must adapt on two fronts: strengthening traditional SEO and embracing <u><a href="https://prerender.io/blog/how-to-get-indexed-on-ai-platforms/">AI optimization</a></u> (often called AEO/GEO). Here are some actionable strategies:</p>
<h3 id="a1u5i" data-block-id="a1u5i">Serve Fully-Rendered Content to Crawlers</h3>
<p data-block-id="78nsc">Many modern ecommerce sites use JavaScript-heavy frameworks that rely on dynamic content rendering, which can prevent AI crawlers from seeing important pages. To avoid that, ensure your pages are pre-rendered or server-side rendered so search engines and AI crawlers can access the full content. Or you can use a tool like <u><a href="https://prerender.io/ai-search/">Prerender.io</a></u> to handle this by serving prerendered HTML versions of your pages directly to bots.</p>
<h3 id="6oa77" data-block-id="6oa77">Allow AI Bot Access in Robots.txt</h3>
<p data-block-id="dk36h">Explicitly add AI crawler user-agents (e.g., GPTBot, ClaudeBot, PerplexityBot, etc.) to your <u><a href="https://prerender.io/blog/robots-txt-for-ecommerce-seo/">robots.txt</a></u> with “Allow” rules.</p>
<h3 id="44f4a" data-block-id="44f4a">Use Structured Data and Answer-Friendly Formats</h3>
<p data-block-id="chhvb">Generative AI tools consume structured data to build knowledge panels or answer snippets. So, add Product, FAQPage, HowTo, and other relevant <u><a href="https://prerender.io/blog/5-types-of-schema-markup-dynamic-websites-should-implement-including-a-tutorial/">schema</a></u> to your pages to help them extract and present your information accurately. Also, write your content or product descriptions in a way that directly and clearly answers users’ queries.</p>
<h3 id="8b9hb" data-block-id="8b9hb">Focus on User Intent and High-Value Queries</h3>
<p data-block-id="82f7t">Bolster your content strategy to cover both informational and transactional/commercial intents to capture buyers at all stages. Also, use long-tail, conversational keywords to match AI and voice search queries.</p>
<h3 id="8hdhn" data-block-id="8hdhn">Embrace Omnichannel Selling and Content</h3>
<p data-block-id="68eih">Make sure your products are listed on all major platforms and integrated with voice assistants where possible. Use high-quality multimedia (images, videos, etc.) to stand out. Also, encourage user reviews as AI crawlers see those social signals as proof of authority.</p>
<h3 id="6s57o" data-block-id="6s57o">Shift Metrics and Attribution</h3>
<p data-block-id="e79vd">With clicks on the decline, track other performance indicators. Use advanced tools to monitor snippet impressions, direct traffic, AI referral traffic, brand search volume, and others.</p>
<h3 id="6q833" data-block-id="6q833">Prepare for AI-Driven Ads and Shopping Features</h3>
<p data-block-id="bup1c">Make sure your shopping feed is optimized (accurate product data, clear images, up-to-date stock) for search bots. Also, be ready to allocate budget for sponsored placements in AI and chat results.</p>
<p data-block-id="7p4mj">Ultimately, the goal is to be present wherever buyers are getting answers and keep your brand top-of-mind, even if they don’t click right away. For more extensive knowledge, check out these <u><a href="https://prerender.io/blog/ecommerce-products-ai-seo/">10 tips to help your product pages show up in AI search tools</a></u>.</p>
<h2 id="cbfi8" data-block-id="cbfi8">How Prerender.io Can Help Your Store Compete in the AI Shopping Era</h2>
<p data-block-id="alk0g">AI shopping is turning visibility is a cutthroat game. Being invisible to AI bots means no traffic. No traffic means no qualified leads, conversions, and ultimately, no revenue.</p>
<p data-block-id="fahfp">To show up on these AI platforms, the quickest, most reliable, and cost-effective solution is using Prerender.io.</p>
<p data-block-id="21cak"><u><a href="https://prerender.io/ai-search/">Prerender.io</a></u> is a JavaScript rendering tool that acts as a headless browser, rendering your pages server-side and serving static HTML to every major bot, including Googlebot, GPTBot, PerplexityBot, and OAI-SearchBot.</p>
<p data-block-id="arbkb">Instead of struggling with dynamic content or infinite-scroll sections, these bots receive a clean, fully loaded version of your site, complete with all the metadata and <u><a href="https://prerender.io/blog/benefits-of-using-open-graph/">Open Graph tags</a></u> that drive visibility.</p>
<p data-block-id="55e2s">In other words, your entire product catalog becomes more discoverable to AI search engines and more likely to appear in zero-click searches. Here’s a graph of one Prerender.io client that saw steady traffic growth within weeks of adding a ChatGPT integration to their account.</p>
<figure class="image strchf-type-image regular strchf-size-regular strchf-align-center"><picture><source srcset="https://images.storychief.com/account_57345/graph-showcasing-ai-platform-traffic-growth-since-implementing-prerenderio_4fe72dd5faf871662cc53d09f7c75e1d_800.png 1x" media="(max-width: 768px)" /><source srcset="https://images.storychief.com/account_57345/graph-showcasing-ai-platform-traffic-growth-since-implementing-prerenderio_4fe72dd5faf871662cc53d09f7c75e1d_800.png 1x" media="(min-width: 769px)" /><img decoding="async" src="https://images.storychief.com/account_57345/graph-showcasing-ai-platform-traffic-growth-since-implementing-prerenderio_4fe72dd5faf871662cc53d09f7c75e1d_800.png" alt="Graph showcasing AI platform traffic growth since implementing Prerender.io." /></picture></figure>
<p data-start="81" data-end="361">But visibility is only half the story. Understanding how crawlers interact with your site is what gives you a real competitive edge.</p>
<p data-start="1317" data-end="1551">Knowing that your site is crawlable is one thing; knowing <em data-start="1374" data-end="1379">who</em> is crawling it is even more powerful. With the Crawler Type view in Prerender.io’s AI Dashboard, you can now see crawler activity grouped by category, including:</p>
<ul data-start="1553" data-end="1766">
<li data-start="1553" data-end="1602">
<p data-start="1555" data-end="1602"><strong data-start="1555" data-end="1573">Search Engines</strong> (Googlebot, Bingbot, etc.)</p>
</li>
<li data-start="1603" data-end="1661">
<p data-start="1605" data-end="1661"><strong data-start="1605" data-end="1620">AI Crawlers</strong> (GPTBot, PerplexityBot, OAI-SearchBot)</p>
</li>
<li data-start="1662" data-end="1719">
<p data-start="1664" data-end="1719"><strong data-start="1664" data-end="1685">Social Media Bots</strong> (Facebook, LinkedIn, X/Twitter)</p>
</li>
<li data-start="1720" data-end="1766">
<p data-start="1722" data-end="1766"><strong data-start="1722" data-end="1735">SEO Tools</strong> (Ahrefs, SEMrush, Moz, etc.)</p>
</li>
</ul>
<p data-start="1768" data-end="2087">This visibility makes it easier to understand your traffic sources, identify which crawlers matter most for your store, and fine-tune your optimization strategy to match. Whether you’re targeting Google rankings, AI shopping results, or social visibility, you’ll know exactly where your products are getting exposure.</p>
<p data-block-id="1qu4e"><img loading="lazy" decoding="async" class="aligncenter wp-image-6046 size-large" src="https://prerender.io/wp-content/uploads/prerender-crawler-type-2-1024x339.png" alt="types of crawlers in the new prerender.io dashboard (social media, AI crawlers, seo, search tools)" width="640" height="212" srcset="https://prerender.io/wp-content/uploads/prerender-crawler-type-2-1024x339.png 1024w, https://prerender.io/wp-content/uploads/prerender-crawler-type-2-300x99.png 300w, https://prerender.io/wp-content/uploads/prerender-crawler-type-2-768x254.png 768w, https://prerender.io/wp-content/uploads/prerender-crawler-type-2-1536x508.png 1536w, https://prerender.io/wp-content/uploads/prerender-crawler-type-2-2048x678.png 2048w" sizes="(max-width: 640px) 100vw, 640px" /></p>
<p data-block-id="1qu4e">By using Prerender.io, you never have to worry about crawl barriers or other technical AI SEO issues. And the icing on the cake is that you can <u><a href="https://prerender.io/pricing/">get started with Prerender.io for free</a></u>.</p>
<h2 id="fo7mv" data-block-id="fo7mv">Elevate Your Ecommerce Store’s Search Presence for AI Shopping</h2>
<p data-block-id="7lce6">Throughout this blog, we’ve pointed out how AI is disrupting customers’ journeys. Rather than resisting these changes, the smartest move is to adapt your strategy early and meet shoppers wherever they’re searching.</p>
<p data-block-id="7g0q1">And alongside the strategies outlined above, Prerender.io makes that adaptation seamless, helping your store remain discoverable to AI crawlers and in AI results without a complete overhaul of your tech stack. It also improves page load speeds, optimizes indexing, and helps your product listings stand out in search results.</p>
<p data-block-id="bsp93">If your goal is to future-proof and scale your visibility as AI shopping evolves, Prerender.io is your low-effort, high-impact solution. <u><a href="https://prerender.io/pricing/">Try Prerender.io today</a></u> to stay ahead in the visibility game.</p>
<p><!-- strchf script --><script>if(window.strchfSettings === undefined) window.strchfSettings = {};window.strchfSettings.stats = {url: "https://prerender.storychief.io/en/ai-shopping-experience?id=1561012310&type=2",title: "Zero Click Searches in Ecommerce: How to Adapt to the New AI Shopping Experience",siteId: "20338",id: ""};(function(d, s, id) {var js, sjs = d.getElementsByTagName(s)[0];if (d.getElementById(id)) {window.strchf.update(); return;}js = d.createElement(s); js.id = id;js.src = "https://d37oebn0w9ir6a.cloudfront.net/scripts/v0/strchf.js";js.async = true;sjs.parentNode.insertBefore(js, sjs);}(document, 'script', 'storychief-jssdk'))</script><!-- End strchf script --></p>
]]></content:encoded>
</item>
<item>
<title>GEO Audit vs. SEO Audit: What’s the Difference for Site Health Checks</title>
<link>https://prerender.io/blog/seo-audit-vs-geo-audit-for-site-health/</link>
<dc:creator><![CDATA[Prerender]]></dc:creator>
<pubDate>Wed, 09 Jul 2025 10:20:30 +0000</pubDate>
<category><![CDATA[AI SEO]]></category>
<category><![CDATA[technical seo]]></category>
<guid isPermaLink="false">https://prerender.io/?p=5863</guid>
<description><![CDATA[Find out what SEO and GEO audits involve and how to improve your site's visibility.]]></description>
<content:encoded><![CDATA[
<p>A few years ago, running an SEO audit was the gold standard for checking your website’s health. You’d run a crawl, spot technical or content issues, and fix them, all to help Google properly rank your content.</p>
<p>But today, as AI-powered search grows alongside traditional search engines, site health checks now need an additional, important lens: a GEO (Generative Engine Optimization) audit.</p>
<p><span style="box-sizing: border-box; margin: 0px; padding: 0px;">Where an SEO audit asks, “<strong>Can Google crawl and rank this page?</strong>”, a GEO audit asks, “<strong>Do generative search engines understand and cite this page in their responses?</strong>”</span></p>
<p>In this article, we’ll break down the differences and similarities between SEO and GEO audits, explain what each one entails, recommend the best tools, and give you a checklist to run both audits with confidence.</p>
<h2 class="wp-block-heading">What is an SEO Audit?</h2>
<p>An SEO audit is essentially a website health check for traditional search. It is a process of <strong>evaluating how well your website is optimized for traditional search engines</strong> like Google.</p>
<p>From speed to structure, the goal of an SEO health check is to ensure that your site follows the best practices and to identify and <a href="https://prerender.io/blog/technical-seo-issues/">fix any issues</a> that may hurt your search ranking.</p>
<h2 class="wp-block-heading">What is a GEO Audit?</h2>
<p>A GEO (Generative Engine Optimization) audit is a newer concept focused on AI-driven search. It examines <strong>how often your site or content is cited, summarized, or linked in responses generated by large language models</strong> (LLMs) like Google Gemini, ChatGPT, and Claude.</p>
<p>And unlike SEO, a GEO audit’s goal is to optimize your content for generative search tools, so your site is the one presented when responding to user prompts.</p>
<p>Not sure if it’s time for an audit? <a href="https://prerender.io/blog/free-site-audit/">Here are six signs your site might need one</a>.</p>
<h2 class="wp-block-heading">Key Components: SEO Audit vs. GEO Audit</h2>
<p>Before running a proper site health check, it’s important to understand what it covers and what to look for. Let’s break down the key components of both SEO and GEO audits.</p>
<p>But first, remember that SEO and GEO audits aren’t two completely different practices. GEO optimization (also called AEO optimization) is built on the same foundation as SEO. The difference is that GEO emphasizes certain SEO elements and adapts them to align with how large language models (LLMs) and AI-driven search engines evaluate content.</p>
<figure class="wp-block-image size-large"><img loading="lazy" decoding="async" width="1024" height="576" src="https://prerender.io/wp-content/uploads/SEO-vs.-GEO-Audit-Checklist-by-Prerender.io-1-1024x576.png" alt="How to check SEO and GEO health - a checklist by Prerender.io" class="wp-image-6217" srcset="https://prerender.io/wp-content/uploads/SEO-vs.-GEO-Audit-Checklist-by-Prerender.io-1-1024x576.png 1024w, https://prerender.io/wp-content/uploads/SEO-vs.-GEO-Audit-Checklist-by-Prerender.io-1-300x169.png 300w, https://prerender.io/wp-content/uploads/SEO-vs.-GEO-Audit-Checklist-by-Prerender.io-1-768x432.png 768w, https://prerender.io/wp-content/uploads/SEO-vs.-GEO-Audit-Checklist-by-Prerender.io-1-1536x864.png 1536w, https://prerender.io/wp-content/uploads/SEO-vs.-GEO-Audit-Checklist-by-Prerender.io-1.png 1920w" sizes="(max-width: 1024px) 100vw, 1024px" /></figure>
<h2 class="wp-block-heading">SEO Audit Checks</h2>
<p>A solid SEO audit has many moving parts, but it can be categorized into three main areas that work together to help your site rank and stay competitive: technical, on-page, and off-page SEO. Let’s look at each of them:</p>
<h3 class="wp-block-heading">1. Technical SEO</h3>
<p>This forms the foundation of your website’s performance in search—without it, nothing else works. Technical SEO focuses on how search engines crawl, render, and index your pages. You’ll check:</p>
<ul class="wp-block-list">
<li><strong>Site speed:</strong> are there slow-loading pages or JS-heavy content?</li>
<li><strong>Mobile-friendliness:</strong> does your site perform just as well on <a href="https://prerender.io/blog/how-to-make-a-website-mobile-friendly/">mobile </a>as desktop?</li>
<li><strong>Crawlability:</strong> are your pages accessible to bots, or are you accidentally blocking them?</li>
<li><strong>Indexing issues:</strong> are important pages showing up in Google’s index? Or are there <a href="https://prerender.io/blog/how-to-fix-duplicate-content-issues/">duplicate pages</a>, or worse, nothing indexed at all?</li>
</ul>
<h3 class="wp-block-heading">2. On-Page SEO</h3>
<p>This part of the audit focuses on making sure your content clearly communicates its purpose to both users and search engines. On-page SEO includes:</p>
<ul class="wp-block-list">
<li><strong>Title tags and meta descriptions:</strong> are they compelling, keyword-aligned, and informative?</li>
<li><strong>Headers (H1, H2, etc.):</strong> is the content scannable and logically organized?</li>
<li><strong>Schema markup: </strong>are you using <a href="https://prerender.io/blog/structured-data-for-seo/">structured data</a> to help search engines generate rich results?</li>
<li><strong>Keyword targeting:</strong> are you using the right terms your audience is searching for?</li>
</ul>
<h3 class="wp-block-heading">3. Off-Page SEO</h3>
<p>This area evaluates how much external trust and authority your site has built across the web. It includes:</p>
<ul class="wp-block-list">
<li><strong>Backlinks:</strong> are you earning backlinks from credible sources?</li>
<li><strong>Domain authority:</strong> how competitive is your site in your niche?</li>
<li><strong>Brand mentions:</strong> is your content being referenced or shared beyond your domain?</li>
</ul>
<h2 class="wp-block-heading">GEO Audit Checks</h2>
<p>A GEO audit builds on traditional SEO best practices but fine-tunes them for generative search. That means the following checks apply to both SEO and GEO audits, while technical SEO, on-page SEO, and off-page SEO remain critical for measuring how your site performs in AI-powered search results.</p>
<h3 class="wp-block-heading">1. Content Structure and Semantic Clarity</h3>
<p>Since LLMs retrieve relevant, well-structured answers, this audit checks that your content is clearly organized and easy to parse</p>
<ul class="wp-block-list">
<li><strong>Smart headings:</strong> are you using headings that mirror how people search or ask questions?</li>
<li><strong>Bullet points and modular chunks:</strong> is your content broken into digestible, quotable sections?</li>
<li><strong>Semantic HTML:</strong> are you using tags like , , and to help models interpret the page layout?</li>
</ul>
<h3 class="wp-block-heading">2. Factual Accuracy</h3>
<p>AI models don’t point to a list of blue links, instead, they generate instant answers. This means they need high-confidence content. So your GEO audit should check for:</p>
<ul class="wp-block-list">
<li><strong>Source credibility:</strong> are you linking to reputable, up-to-date sources?</li>
<li><strong>Date stamps and authorship:</strong> is your content high-quality, constantly updated, and credited to real people?</li>
<li><strong>Verified statements:</strong> are you backing up your statements with evidence and citations?</li>
</ul>
<h3 class="wp-block-heading">3. Entity Clarity and Brand Recognition</h3>
<p>To show up in AI-generated answers, LLMs need to recognize your brand as a reliable entity. This, of course, starts with clear and consistent branding across your site.</p>
<ul class="wp-block-list">
<li><strong>Consistent naming:</strong> are your bios, pages, and structured data aligned with the same brand identifiers?</li>
<li><strong>Trust signals:</strong> do you have an about page, author bios, and contact info that show legitimacy?</li>
<li><strong>Organization and person schema:</strong> are you using organization and person schema to help models connect the dots between your brand and your expertise?</li>
</ul>
<h3 class="wp-block-heading">4. Topical Authority and Depth</h3>
<p>GEO audits care more about your coverage of a topic than keyword variety. This component checks that your site provides comprehensive, well-connected content on a subject. It checks for:</p>
<ul class="wp-block-list">
<li><strong>Content clusters: </strong>are you creating related pages that link together and support each other?</li>
<li><strong>Follow-up questions:</strong> are you addressing the full range of what a user might ask?</li>
<li><strong>Comprehensive posts:</strong> are you offering real, in-depth explanations—not just surface-level content?</li>
</ul>
<h3 class="wp-block-heading">5. Technical Health</h3>
<p>While a GEO/AI SEO audit is mostly content-focused, technical performance still plays a supporting role. After all, even the best content needs to be seen to be cited. So, this audit verifies that all important content is visible to <a href="https://prerender.io/blog/how-to-optimize-your-website-for-ai-crawlers/">AI crawlers</a> by checking for:</p>
<ul class="wp-block-list">
<li><strong>Fast-loading pages:</strong> AI-powered tools don’t wait around for slow servers.</li>
<li><strong>Bot accessibility:</strong> to prevent unintentional blockage of AI crawlers.</li>
<li><strong>Readable, clean HTML:</strong> this improves how your content is parsed and understood.</li>
<li><strong>JavaScript prerendering:</strong> to ensure that your JS-powered content is available to crawlers on request.</li>
</ul>
<h2 class="wp-block-heading">Recommended Tools for SEO and GEO Audits</h2>
<p>Now that you know what to look for in both SEO and GEO audits, how can you actually check them?</p>
<p>Well, one powerful JavaScript SEO solution that pulls double duty is <strong>Prerender.io</strong>. With its <a href="https://prerender.io/resources/free-downloads/site-audit-tool/">free site audit tool</a>, you can easily assess how well search and AI crawlers can access your content.</p>
<p>In addition, Prerender.io diagnoses and helps resolve technical issues by serving pre-rendered HTML for your dynamic pages, ensuring they are immediately crawlable and indexable. It also improves site speed for bots, conserves your <a href="https://prerender.io/resources/free-downloads/site-audit-tool/">crawl budget</a>, and reduces the risk of your pages being skipped or partially indexed.</p>
<p>See how Haarshop achieved a <strong>99 PageSpeed score and saw a 50% traffic increase</strong> after adopting Prerender.io. <a href="https://prerender.io/resources/case-studies/improved-pagespeed-and-boosting-page-indexing-to-optimize-webshop/">Read the full case study here</a>.</p>
<figure class="wp-block-image size-large"><img loading="lazy" decoding="async" width="1024" height="580" src="https://prerender.io/wp-content/uploads/Haarshops-50-traffic-increase-after-using-Prerender.io--1024x580.png" alt="Haarshop experienced a 50% traffic increase after using Prerender.io" class="wp-image-5869" srcset="https://prerender.io/wp-content/uploads/Haarshops-50-traffic-increase-after-using-Prerender.io--1024x580.png 1024w, https://prerender.io/wp-content/uploads/Haarshops-50-traffic-increase-after-using-Prerender.io--300x170.png 300w, https://prerender.io/wp-content/uploads/Haarshops-50-traffic-increase-after-using-Prerender.io--768x435.png 768w, https://prerender.io/wp-content/uploads/Haarshops-50-traffic-increase-after-using-Prerender.io-.png 1200w" sizes="(max-width: 1024px) 100vw, 1024px" /></figure>
<p>That said, below are other tools—ranging from free to paid—that can help diagnose and improve your site’s performance for traditional and AI search engines.</p>
<p><strong>Note:</strong> This is not an exhaustive list. We’re only listing a handful of tools covering the key components discussed in the section above.</p>
<h3 class="wp-block-heading">Tools For SEO Audits</h3>
<figure class="wp-block-table"><table class="has-fixed-layout"><thead><tr><th>Tool</th><th>SEO Application</th><th>Pricing</th></tr></thead><tbody><tr><td>1. Screaming Frog</td><td>To crawl your site and identify technical SEO issues.</td><td>Starts at $259/year</td></tr><tr><td>2. Ahrefs</td><td>To evaluate keyword rankings, backlink health, and expose content gaps.</td><td>Starts at $99/month</td></tr><tr><td>3. Google Search Console</td><td>To see indexing errors, performance reports, and structured data issues.</td><td>Free</td></tr><tr><td>4. W3C Validator</td><td>To ensure correct HTML and markup validation.</td><td>Free</td></tr><tr><td>5. PageSpeed Insights</td><td>To evaluate load speed and <a href="https://prerender.io/blog/how-to-pass-core-web-vitals/">Core Web Vitals</a> for user experience and rankings.</td><td>Free</td></tr></tbody></table></figure>
<p><strong>Resource:</strong> Need more recommendations? <a href="https://prerender.io/blog/best-technical-seo-tools/">See our top 10 technical SEO auditing tools</a>.<br></p>
<h3 class="wp-block-heading">Tools For GEO Audits</h3>
<figure class="wp-block-table"><table class="has-fixed-layout"><thead><tr><th>Tool</th><th>GEO Application</th><th>Pricing</th></tr></thead><tbody><tr><td>1. ChatGPT, Perplexity, etc.</td><td>To gauge content visibility and citation in AI-generated answers.</td><td>Varies with each tool</td></tr><tr><td>2. Frase</td><td>To highlight FAQ and summary formatting aligned with LLM output patterns.</td><td>Starts at $45/month</td></tr><tr><td>3. Keyword.com</td><td>To track AI rankings, keywords, and brand mentions on AI search platforms, including ChatGPT, Google AI Overviews, and Perplexity.</td><td>Starts at $24.5/month</td></tr><tr><td>4. Google Rich Results Test</td><td>To ensure structured data is valid for featured snippets and generative summaries.</td><td>Free</td></tr><tr><td>5. InLinks</td><td>To identify entity gaps and suggest schema and internal links for clearer AI context.</td><td>Starts at $49/month</td></tr><tr><td>6. MarketMuse</td><td>To evaluate content depth and semantic relevance using AI ranking factors.</td><td>Starts at $129/month</td></tr></tbody></table></figure>
<p><strong>Resource</strong>: <a href="https://prerender.io/blog/best-technical-geo-and-ai-se-tools/">Check out our top GEO tools for AI search optimization</a>.</p>
<h2 class="wp-block-heading">What LLMs Prioritize When Surfacing Content (and What They Don’t)</h2>
<p>Even though both traditional and AI search engines aim for content visibility, the way they surface content is fundamentally different. Since LLMs are the new kids on the block, they’ll be our focus here.</p>
<p>Not sure how Google ranks content? <a href="https://prerender.io/blog/the-most-important-ranking-factors-for-2023/">Here’s a refresher on traditional SEO</a>.</p>
<p>At a high-level, LLMs prioritize content that’s clear, well-structured, and context-rich. They favor conversational phrasing, helpful summaries, and topic clarity over things like keyword usage and density. If your content includes original research, statistics, or well-cited insights, it’s an added trust signal to getting referenced in AI-generated responses.</p>
<p>On the flip side, traditional SEO signals like meta tags, link anchors, or user experience signals tend to matter less in LLM-driven environments.</p>
<p>This prioritization is one of the reasons why a GEO audit is important: to ensure your content is optimized for AI models and potentially included in their answers. See our full comparison of <a href="https://prerender.io/blog/seo-vs-aio-vs-geo/">SEO vs. AIO vs. GEO</a> to better understand the new search landscape.</p>
<p>For more in-depth information and guide on your LLM optimization journey, check out these blogs:</p>
<ul class="wp-block-list">
<li><a href="https://prerender.io/blog/ai-optimization-technical-seo-guide/">AI SEO Optimization: A Technical SEO Guide</a>.</li>
<li><a href="https://prerender.io/blog/best-technical-geo-and-ai-se-tools/#elementor-toc__heading-anchor-3">How to Position Your Website for GEO and AI Search Success</a>.</li>
<li><a href="https://prerender.io/blog/ecommerce-products-ai-seo/">10 Tips to Help Your Products Show Up in ChatGPT</a>.</li>
</ul>
<h2 class="wp-block-heading">How to Run a Technical SEO Audit and a Generative Engine Optimization Audit</h2>
<p>If you’re ready to audit your site from both angles, here’s a brief rundown of what each auditing process looks like:</p>
<h3 class="wp-block-heading">Running an SEO Audit:</h3>
<ul class="wp-block-list">
<li><strong>Crawl your site</strong> with Screaming Frog to get a full snapshot of how search engines see your pages.</li>
<li><strong>Use Google Search Console</strong> for crawl errors, SEO indexing issues, and performance insights.</li>
<li><strong>Review URL structure and internal</strong> linking for content clusters.</li>
<li><strong>Audit outdated, thin, or duplicate content</strong> that dilutes topical relevance.</li>
<li><strong>Evaluate technical health</strong> by checking Core Web Vitals, mobile responsiveness, HTTPS, and robots.txt.</li>
<li><strong>Analyze your backlink profile</strong> using Ahrefs or Semrush, then prioritize fixes based on impact and performance gaps.</li>
</ul>
<p><strong>Related</strong>: <a href="https://prerender.io/blog/how-to-conduct-a-technical-seo-audit/">Read our guide to successful technical SEO audits</a>.</p>
<h3 class="wp-block-heading">Running a GEO Audit:</h3>
<ul class="wp-block-list">
<li><strong>Search your brand</strong> in tools like Gemini, ChatGPT, or Perplexity to check if your entity is properly identified.</li>
<li><strong>Test LLM prompts</strong> in your niche to see who the models cite—you or your competitors?</li>
<li><strong>Use Surfer SEO or MarketMuse</strong> to analyze your content’s structure and semantic clarity.</li>
<li><strong>Validate schema markup</strong> with Google Rich Results Test to ensure your structured data is readable and complete.</li>
<li><strong>Benchmark your visibility</strong> against competitors using Ahrefs or Semrush. Compare things like snippet ownership, domain authority, and content freshness.</li>
<li><strong>Create a GEO scorecard</strong> to evaluate your site’s visibility and overall performance in generative tools. This will help you track improvement over time.</li>
</ul>
<h2 class="wp-block-heading">Staying Search-Ready Means Running Both SEO and GEO Audits</h2>
<p>It’s no secret that <a href="https://www.semrush.com/blog/ai-search-report/" target="_blank" rel="noreferrer noopener nofollow">search behavior has changed</a>. As many users still click search links, many others are turning to AI for instant answers. That’s why it’s no longer enough to optimize just for Google. You also need to make sure your content is optimized for and <a href="https://prerender.io/blog/how-to-get-indexed-on-ai-platforms/">discoverable by AI search engines</a>.</p>
<p>An SEO audit will help you stay compliant with technical and on-page best practices, while a GEO audit ensures your content gets cited, summarized, or linked in generative answers. Both audits are complementary and essential to your site’s health and visibility.</p>
<p>And if your site runs on JavaScript, a tool like <strong>Prerender.io</strong> can confidently power both audits and generative engine optimization efforts by making your content accessible to bots and AI crawlers.</p>
<p>With the new <strong>AI Dashboard</strong>, you can filter crawler activity by type—whether it’s search engines, social platforms, AI crawlers, or SEO tools—and see exactly how each is interacting with your site. This visibility helps you understand which crawlers matter most, what pages they visit, and where to focus your optimization efforts.</p>
<p>In other words, you’re not just preparing your site for today’s search engines—you’re future-proofing it for AI-driven discovery. <a href="https://prerender.io/pricing/">Give Prerender.io</a> a try to future-proof your AI search visibility.</p>
<figure class="wp-block-image size-full"><img loading="lazy" decoding="async" width="1024" height="545" src="https://prerender.io/wp-content/uploads/Screen-Shot-2025-08-18-at-10.05.54-AM-1024x545-1.png" alt="AI crawler tracking on Prerender.io" class="wp-image-6191" srcset="https://prerender.io/wp-content/uploads/Screen-Shot-2025-08-18-at-10.05.54-AM-1024x545-1.png 1024w, https://prerender.io/wp-content/uploads/Screen-Shot-2025-08-18-at-10.05.54-AM-1024x545-1-300x160.png 300w, https://prerender.io/wp-content/uploads/Screen-Shot-2025-08-18-at-10.05.54-AM-1024x545-1-768x409.png 768w" sizes="(max-width: 1024px) 100vw, 1024px" /></figure>
<h2 class="wp-block-heading">FAQs About Technical and AI SEO Audits</h2>
<p>Let’s review some of the most common questions about technical SEO audits and newer generative engine optimization audits.</p>
<h3 class="wp-block-heading">1. What’s the Difference Between an SEO Audit and a GEO Audit?</h3>
<p>An SEO audit focuses on how traditional search engines like Google crawl, index, and rank your site. A GEO audit (Generative Engine Optimization audit) checks how well your content performs in AI search tools like ChatGPT or Google Gemini, ensuring your pages are cited or summarized in LLM-generated results.</p>
<h3 class="wp-block-heading">2. How Do I Run a Technical SEO Audit for Website Health?</h3>
<p>To run a technical SEO audit, start by crawling your site using tools like Screaming Frog or Prerender.io’s free site audit tool. Look for issues with page speed, crawlability, mobile-friendliness, and indexing. Fixing these technical gaps ensures your content can be discovered and ranked by traditional search engines—and also supports your GEO audit by making content accessible to AI crawlers.</p>
<h3 class="wp-block-heading">3. How Can I Optimize My Website for AI Search Engines and LLMs?</h3>
<p>You can optimize for AI search engines by running a GEO audit. This includes structuring your content semantically, improving factual accuracy, using schema markup, and building topical authority so LLMs recognize and reference your brand in generative answers.</p>
]]></content:encoded>
</item>
</channel>
</rss>