Google’s 15MB crawl limit is a crucial factor to consider when optimizing your website.
But what exactly does this limit mean?
And how does it affect your website’s performance in search results?
In this post, we’ll delve into these questions and more, providing you with a comprehensive understanding of Google’s 15MB crawl limit and how to effectively manage it.
What is Google’s 15MB Crawl Limit?
This limit refers to the maximum size of a web page that Googlebot will download and process for indexing. If a web page exceeds this limit, Googlebot will only process the first 15MB of the page.
It is applied per resource basis, meaning that each HTML file, JavaScript file, and CSS file on your site has its own 15MB limit. However, despite each resource having its own limit, you still need to ensure your money-maker or “golden egg” content is considered. For JavaScript-heavy websites, on the other hand, the impact is even bigger. Naturally, JS files are large and can quickly exceed the 15MB limit.
The primary reason for Google implementing this rule is to manage the resources used during the crawling and indexing process. While it helps Googlebot crawl and index the vast number of web pages on the internet, it’s not always a win for your site.
Is the 15MB Limit the Same as Crawl Budget?
The limit is separate from but related to Google’s crawl budget. Your crawl budget refers to the number of pages Googlebot can crawl on your site within a certain timeframe. If a page is close to or exceeds the 15MB limit, Googlebot may use up more of your allocated crawl budget to download and process that page. This leaves fewer resources for crawling other pages on your site.
Related: How to Guide Googlebot in Crawling Important URLs
The limit includes all resources that are embedded in the HTML of a webpage. This includes text content, HTML markup, CSS, and JavaScript. External resources like images and videos are not counted towards the 15MB limit unless their data is embedded directly in the HTML.
While images and videos are not counted towards the 15MB limit, large images and videos can still impact a page’s loading time, which can affect Googlebot’s ability to efficiently crawl the page.
Related: PageSpeed Explained
Is it possible to hit a 15MB file size for HTML pages? For most websites, it is not reasonable or necessary to have HTML pages that approach or exceed the 15MB limit. Most web pages are far smaller than this limit. However, JavaScript websites or JS-based elements pages can exceed this limit.
Strategies and Techniques to Avoid the 15MB Limit
There are several strategies you can employ to avoid this limit. Some of them include optimizing your HTML, CSS, and JavaScript files to reduce their size, using external resources instead of embedding large amounts of data within your HTML, and implementing techniques like lazy loading for images and videos. Let’s go into detail.
1. Server-Side Rendering (SSR)
Server-side rendering (SSR) can be used to process JavaScript; serving crawlers a fully rendered HTML version. Additionally, server-side optimizations can include techniques like code minification and compression, which reduces the size of HTML, CSS, and JavaScript files.
However, it’s important to note that server-side rendering is not the optimal choice for every website.
SSR requires a significant amount of server resources. This can lead to increased server load and potentially slower response times, especially for websites with heavy traffic or complex JavaScript applications. Additionally, implementing SSR can be a costly, complex process that requires significant changes to your website’s architecture and codebase.
Dynamic rendering offers similar benefits at a fraction of the cost. A tool like Prerender, for example, helps Google to easily crawl and index a website by generating a static HTML version of each page.
Related: When You Should Consider Dynamic Rendering
Determining and Tracking Your Website’s Size
You can determine the size of your website using various tools and techniques.
One common method is to use an auditing tool to crawl your site and provide information about the size of each page. You can also manually check the size of your HTML, CSS, and JavaScript files.
Google Search Console provides detailed information about how Googlebot interacts with your site. Other tools like Screaming Frog can mimic the behavior of web crawlers, allowing you to diagnose potential issues.
Make Use of Embedded or Linked SVGs
Including SVGs as image tags can help manage the page’s size, as the data for the image is not embedded in the HTML. However, this can increase the number of HTTP requests the page makes, which can impact load time. The best approach depends on the needs and constraints of your website.
Final Thoughts
In addition to the 15MB limit, increasing your crawl budget will ensure your most important pages get crawled and indexed by Google every time.
Struggling to get indexed? Get started with 1,000 URLs free.
