If you’re in the SEO world, you’ve probably seen the latest discourse about AI crawlers and JavaScript. The verdict: AI crawlers can’t effectively render JavaScript.
As LLMs start to heavily influence the SEO scene, this could be another uphill battle for SEO teams and developers alike.
But, we have good news. There’s an easy fix to help AI crawlers render your JavaScript pages: a simple installation with Prerender.io. The image below speaks for itself—see the traffic levels after a client added the ChatGPT user agent to their Prerender account.
Curious to learn more? Let’s dive into the details.
First: Understanding the Impact of AI Crawlers
As you’re aware, AI crawlers are the next generation of web indexing tools. Unlike traditional search crawlers, which primarily focus on parsing HTML and following links, AI crawlers use advanced language models to understand and contextualize web content.
This allows LLMs like ChatGPT, Claude, Perplexity, and more to grasp the meaning and relevance of content more like a human. They can understand natural language, interpret context, and even make inferences based on the content they encounter. Users get more accurate search results and content recommendations.
More and more users are leaning towards LLMs as their search engine of choice. This is having a significant impact in the SEO world. LLMs are starting to gain greater market share. Results from a recent study shows that AI crawlers now account for ~28% of Googlebot’s traffic volume, and this is only expected to rise.
Search Engine Journal cited the study: “OpenAI’s GPTBot and Anthropic’s Claude generate nearly 1 billion requests monthly across Vercel’s network… GPTBot made 569 million requests in the past month, while Claude accounted for 370 million. Additionally, PerplexityBot contributed 24.4 million fetches, and AppleBot added 314 million requests.”
While it’s roughly 28% of Googlebot’s volume now, we can only expect these numbers to rise in the months and years to come.
Why Do AI Crawlers Struggle to Render JavaScript?
But why do AI crawlers struggle with JavaScript rendering? Like traditional search engines, it’s tough for them to render JS for a few reasons.
Static vs. Dynamic Content
JavaScript often generates content dynamically on the client-side, which occurs after the initial HTML has loaded. This means that when an AI crawler reads the page, it may only see the initial HTML structure without the rich, dynamic content that JavaScript would typically load for a human user. Depending on your website, this can lead to a major discrepancy in what content AI is pulling from your site versus what human users see.
Related: Traditional Search vs. AI-Powered Search Explained
Execution Environment
AI crawlers typically don’t have a full browser environment to execute JavaScript.
They lack the JavaScript engine and DOM (Document Object Model) that browsers use to render pages. Without these components, the crawler can’t process and render JavaScript in the same way a web browser would. This limitation is particularly problematic for websites that rely heavily on JavaScript frameworks like React, Angular, or Vue.js, where much of the content and functionality is generated through JavaScript.
Asynchronous Loading
Many JavaScript applications load content asynchronously, which can be problematic for crawlers operating on a set timeframe for each page. Crawlers might move on to the next page before all asynchronous content has loaded, missing critical information. This is particularly challenging if your website uses infinite scrolling or lazy loading techniques.
Single Page Applications (SPAs)
AI crawlers may struggle with the URL structure of SPAs, where content changes without corresponding URL changes. In SPAs, navigation often occurs without page reloads, using JavaScript to update the content. This can confuse crawlers, which traditionally rely on distinct URLs to identify different pages and content. As a result, much of an SPA’s content might be invisible to AI crawlers, appearing as a single page with limited content.
Related: How to Optimize your Single-Page App for SEO
Security Limitations
For security reasons, AI crawlers are often restricted in their ability to execute external scripts, which is a common feature in many JavaScript-heavy sites. This limitation is designed to protect against potential security threats, but it also means that content loaded from external sources or generated by third-party scripts may not be visible to the crawler. This can be particularly problematic for sites that rely on external APIs or services to generate dynamic content.
Related: How to Manage Dynamic Content
How AI Crawlers’ Limitations Affect Your SEO
While many search engines struggle to render JavaScript anyway, the impacts for AI crawlers’ inefficiencies may, eventually, be more impactful. Any content loaded dynamically through JavaScript may be completely invisible to AI crawlers.
This can have hard-hitting results on your SEO and your bottom line. For example, Ahrefs recently stated that they’ve received 14,000+ customers from ChatGPT alone. If your content isn’t being featured in LLMs, this could wreak havoc on the discoverability of your site.
Some websites may be worse than others, particularly those that may struggle with an inefficient crawl budget. Factors include:
- Site size. Larger sites can face crawling inefficiencies as more resources are required to crawl them.
- Sites with frequently changing content. If your content is updated on a daily—or even weekly—basis, you’ll burn through your crawl budget more quickly and your pages may not be crawled as quickly as you’d like.
- Sites with slow rendering speeds. Slow or error-prone sites may see reduced crawling by certain bots, and therefore, could be impacted by poor JS rendering from AI crawlers, too.
Does your site check off any of the above boxes? If so, you might need some extra support.
A Solution to Help AI Crawlers Render Your Site: Prerender.io
Prerender.io is a dynamic rendering solution that helps bots efficiently crawl your pages. This can help with structured data enhancement, rapid indexing improvements, organic traffic levels, improved social sharing previews, and more.
Prerender works with all major bots and crawlers. Since AI crawlers came on the scene, our data is showing positive results, too. For example, the image below shows traffic levels from one Prerender customer after adding the ChatGPT user agent to their account.
How Does Prerender.io Work?
Put simply, we convert your dynamic JavaScript pages into static HTML for crawlers to read quickly. Here’s an overview:
- When a crawler visits your site, Prerender intercepts the request.
- It serves a pre-rendered, static HTML version of your page to the crawler.
- This static version contains all the content that would normally be loaded dynamically via JavaScript.
You can also add various user agents to your account, such as ChatGPT-user like the image above. Here’s a list of some of Prerender’s user agents and how to add them to your account.
The Results
From there, you should see:
- Faster rendering times. Crawlers receive instantly loadable HTML, dramatically reducing rendering time.
- Improved crawl efficiency. More of your pages can be crawled within the allocated budget.
- Better indexing rates. Your content, including dynamically loaded content, becomes visible to all available bots.
- Fewer drops in traffic. Poor JS rendering is a lesser-known cause of traffic drops. Clients often come to us after traffic drops from site migrations or algorithm updates for positive results.
For example, here are organic traffic levels from Dutch webshop, Haarshop, after installing Prerender. Read the case study to learn more.
Which Sites Can Benefit From Prerender.io?
There are a few types of websites that can benefit most from Prerender.
- Large sites (1M+ unique pages) with content that changes moderately often, such as once per week. An example might be an enterprise ecommerce store.
- Medium to large sites (10,000+ unique pages) with rapidly changing content, i.e. content that changes on a daily basis. Examples include: job boards, ecommerce stores, gaming and betting platforms, and more.
- Sites with indexing challenges. This is especially if a large portion of your total URLs are classified by GSC as Discovered – currently not indexed.
- Sites with slow rendering problems. Google dislikes pages slower than 20-30 seconds. If your average site rendering time is greater than 30 seconds and less than one minute, Prerender.io is a fantastic option to improve your speeds.
Try Prerender.io For Free
Ready to see if Prerender is right for you? With a 4.7-star Capterra rating and a 75,000+ customer base, Prerender helps JavaScript sites around the world improve their online visibility. Try today with 1000 renders for free.
FAQs
Is Prerender.io Easy To Integrate?
Yes! Integration is simple, and you should be fully integrated within a few hours. Learn more.
Can I Use Prerender.io With Other SEO Techniques?
Absolutely, and you should! Prerender works best when your SEO is in check. Continue optimizing your SEO strategy while using Prerender on the side. Prerender addresses the technical aspect of making your content visible to crawlers. In tandem with effective SEO strategies, you should see strong SEO improvements.
How Does Prerender.io Affect My Site’s User Experience?
Prerender only serves pre-rendered pages to crawlers and bots, not users. Regular users will continue to see and interact with your dynamic, JavaScript-powered site as usual, but behind the scenes, Prerender improves your site’s visibility to search engines and AI crawlers.
How Does Prerender.io Impact My Website’s Loading Speed?
For crawlers, it improves the loading speed significantly by serving pre-rendered HTML content. This can indirectly benefit your SEO, as page speed is a crucial ranking factor for many search engines.
Does Prerender.io Benefit All Websites?
While Prerender helps most JavaScript websites, it’s particularly helpful for:
- Single Page Applications (SPAs)
- Websites with quickly-changing or dynamic content
- E-commerce sites with frequently changing inventory
- News sites or blogs with regular updates
It may not be necessary for simple static websites or those already using server-side rendering. However, even these sites might benefit if they’re looking to optimize for AI crawlers specifically.
