How to Install Prerender (Direct, Ingress & CDN/Cloudflare)

Prerender® is a hugely cost-saving JavaScript SEO option that can save you thousands of dollars and hundreds of hours of manpower compared to other more labor-intensive options. It’s also more reliable and much more likely to earn you top-ranking positions on Google search results.

With that being the case, as with any software it requires an initial set-up process that can take some time to navigate, depending on what your techstack is.

We’ll break down your options in terms of installing Prerender, point out some things to look out for, and outline some suggested steps to take after installation.

Getting Started

Installing Prerender can be broken down into four steps:

1. Identify your techstack and preferred installation method
2. Install the middleware code block appropriate for your tech stack
3. Test to make sure it’s working
4. Configure and customize it

The first thing you’ll need to do is to choose a backend technology stack to integrate with Prerender. This will decide which middleware you use and how you’ll use it.

When Googlebot and other search engine web crawlers make an HTTP request from your website’s server, our middleware will intercede it and deliver a static version of your web page with the extra HTML from executed JavaScript that crawlers can’t usually see. For human users, they’ll be directed as normal down the usual server pathways to your webpage.

The general idea is the same no matter which method you use: you’re going to add code to each page that needs to be pre-rendered. That code contains the API key that was issued to you when you signed up for Prerender.

There are three main installation methods you can choose from:

A. Direct Integration

B. Ingress Type of Integration

C. CDN Integration

Installation Method

We’ll break down the installation methods one by one, the most popular languages used for each, and tell you where to install your API code.

A complete list of Prerender middleware compatible with most popular programming languages can be found here.

Direct Integration

Direct integration involves getting Prerender to work alongside your front-end technology. Commonly this is done using Node.JS or Django/Python.

Middle-ware based direct integrations are almost always more customizable. This is because the logic is in the code itself you can change to suit your needs,  rather than in a static configuration.

Node.JS

Node.JS is an open-source JavaScript framework that executes your content outside of the web browser. It’s primarily used for non-blocking, event-driven servers and is frequently used for traditional websites and back-end API services.

Our Node.JS middleware intercepts HTTP file requests from browsers and then calls Prerender to deliver the static HTML version of your page.

• First, install the Prerender node via npm:

$ npm install prerender-node –save

• Set up the express app:

app.use(require(‘prerender-node’));

OR, if you already have the Prerender.io API token:

app.use(require('prerender-node').set('prerenderToken', 'YOUR_TOKEN'));

You can read the installation instructions for Node.JS here.

Python/Django

Python is a universally known programming language that is often considered to be one of the most useful, as well as one of the easiest programming languages to learn. It’s an all-purpose programming language that emphasizes the importance of code readability and features frequent use of indentation.

1. First, do a PIP install:

pip install django-seo-js

2. Add the pip install to your settings.py file:

# If in doubt, just include both.  Details below.
MIDDLEWARE_CLASSES = (
    'django_seo_js.middleware.EscapedFragmentMiddleware',  # If you're using #!
    'django_seo_js.middleware.UserAgentMiddleware',  # If you want to detect by user agent
) + MIDDLEWARE_CLASSES
INSTALLED_APPS += ('django_seo_js',)
# If you're using prerender.io (the default backend):
SEO_JS_PRERENDER_TOKEN = "123456789abcdefghijkl"  # Really, put this in your env, not your codebase.


3. Finally, add it to your base.html file:

{% load django_seo_js %}
<head>
    {% seo_js_head %}
    ...
</head>

You can read the installation instructions here.

Ingress Installation

Choosing an ingress method of installing Prerender means configuring Prerender to work with your server-side, back-end language, typically by rewriting your .htaccess file. Two of the most common languages used here are Apache and NGINX, but you can find a more complete list of middleware installation instructions here.

Let’s start with Apache.

Apache

Apache is a server-side scripting language commonly used in enterprise web applications. It’s used by roughly 67% of all the web servers in the world. They are easy to customize, fast, reliable, and used by websites that get upwards of hundreds of thousands of sessions per month.

Installing Prerender with Apache involves using the httpd.conf and .htaccess files. View the Apache installation instructions here.

httpd.conf

1. Put this codeblock into your .htaccess file and remove the / comments to allow the use of a proxy over https

LoadModule headers_module libexec/apache2/mod_headers.so
LoadModule proxy_module libexec/apache2/mod_proxy.so
LoadModule proxy_http_module libexec/apache2/mod_proxy_http.so
LoadModule ssl_module libexec/apache2/mod_ssl.so
LoadModule rewrite_module libexec/apache2/mod_rewrite.so

2. Add this to the bottom of the .conf file:

# Prerender.io configuration
# Allow proxying over https
SSLProxyEngine on
# These two commands ignore the validity of the SSL Certificate of the Prerender server
# Only uncomment the two SSLProxy lines if you are testing with a local prerender server over https with a self-signed cert
# A hosted Prerender server should have a valid SSL Certificate so the next two lines can stay commented in production
#SSLProxyCheckPeerName off
#SSLProxyVerify none
.htaccess

This is the actual configuration that sends the request to Prerender®:

# Change YOUR_TOKEN to your prerender token
# Change https://service.prerender.io/ (at the end of the last RewriteRule)
# to http://localhost:3000/ when testing with a local prerender server

<IfModule mod_headers.c>
    RequestHeader set X-Prerender-Token "YOUR_TOKEN"
    RequestHeader set X-Prerender-Version "prerender-apache@2.0.0"
</IfModule>

<IfModule mod_rewrite.c>
    RewriteEngine On

    <IfModule mod_proxy_http.c>
        RewriteCond %{HTTP_USER_AGENT} googlebot|bingbot|yandex|baiduspider|facebookexternalhit|twitterbot|rogerbot|linkedinbot|embedly|quora\ link\ preview|showyoubot|outbrain|pinterest\/0\.|pinterestbot|slackbot|vkShare|W3C_Validator|whatsapp [NC,OR]
        RewriteCond %{QUERY_STRING} _escaped_fragment_
        RewriteCond %{REQUEST_URI} ^(?!.*?(\.js|\.css|\.xml|\.less|\.png|\.jpg|\.jpeg|\.gif|\.pdf|\.doc|\.txt|\.ico|\.rss|\.zip|\.mp3|\.rar|\.exe|\.wmv|\.doc|\.avi|\.ppt|\.mpg|\.mpeg|\.tif|\.wav|\.mov|\.psd|\.ai|\.xls|\.mp4|\.m4a|\.swf|\.dat|\.dmg|\.iso|\.flv|\.m4v|\.torrent|\.ttf|\.woff|\.svg))
        RewriteRule ^(index\.html|index\.php)?(.*) https://service.prerender.io/%{REQUEST_SCHEME}://%{HTTP_HOST}/$2 [P,END]
    </IfModule>
</IfModule>

NGINX

NGINX is a popular alternative to Apache. It has grown in popularity since its release due to its light-weight resource utilization and its ability to easily scale. It’s also useful for serving static content easily and pass requests for dynamic content off to other software – which is exactly what makes it so compatible with Prerender®.

You can read the installation instructions for NGINX here.

The official nginx.conf for Prerender® is below:

# Change YOUR_TOKEN to your prerender token
# Change example.com (server_name) to your website url
# Change /path/to/your/root to the correct value
server {
    listen 80;
    server_name example.com;
    root   /path/to/your/root;
    index  index.html;
    location / {
        try_files $uri @prerender;
    }
    location @prerender {
        proxy_set_header X-Prerender-Token YOUR_TOKEN;   
        set $prerender 0;
        if ($http_user_agent ~* "googlebot|bingbot|yandex|baiduspider|twitterbot|facebookexternalhit|rogerbot|linkedinbot|embedly|quora link preview|showyoubot|outbrain|pinterest\/0\.|pinterestbot|slackbot|vkShare|W3C_Validator|whatsapp") {
            set $prerender 1;
        }
        if ($args ~ "_escaped_fragment_") {
            set $prerender 1;
        }
        if ($http_user_agent ~ "Prerender") {
            set $prerender 0;
        }
        if ($uri ~* "\.(js|css|xml|less|png|jpg|jpeg|gif|pdf|doc|txt|ico|rss|zip|mp3|rar|exe|wmv|doc|avi|ppt|mpg|mpeg|tif|wav|mov|psd|ai|xls|mp4|m4a|swf|dat|dmg|iso|flv|m4v|torrent|ttf|woff|svg|eot)") {
            set $prerender 0;
        }        
        #resolve using Google's DNS server to force DNS resolution and prevent caching of IPs
        resolver 8.8.8.8;
        if ($prerender = 1) {       
            #setting prerender as a variable forces DNS resolution since nginx caches IPs and doesnt play well with load balancing
            set $prerender "service.prerender.io";
            rewrite .* /$scheme://$host$request_uri? break;
            proxy_pass http://$prerender;
        }
       if ($prerender = 0) {
            rewrite .* /index.html break;
        }
    }
}

CDN/CLOUDFLARE

A content delivery network (CDN) is a piece of software used to deliver your content globally across any region of the world using a global network of proxy servers. They’re often used by websites with a global audience to deliver web content at the same speed and performance level. Cloudflare is one of the most popular CDN services, and the one which you’re most likely to configure with Prerender.

The installation process using the CDN/Cloudflare method is fairly straightforward, and you can find it here. This method makes the assumption that you already have Cloudflare installed on your website and have access to its dashboard.

If your website has a CDN, then it is highly recommended that you use this method. Otherwise, the additional caching layer.

Installation

1. Select your website in Cloudflare:

2. Navigate to Workers -> Manage workers -> Create a worker
3. Copy and paste the content of index.js over the code editor on the left hand side, overwriting its current content. Please note the name of the worker for later reference(top-left corner).

4. Replace your API_KEY value with your actual API key from the Prerender dashboard, and edit the PRERENDERED_DOMAINS array to match your site configuration.

5. Click on Save,  Deploy, and navigate to the workers from Step 2. Click on Add Route
6. Edit your route to match your website configuration:

7. Click Save, and you’re all set

Test

At this point you should have Prerender now installed on your website. However, you still need to test it to make sure it’s working properly.

First, open your console and run these commands:

git clone https://github.com/prerender/prerender.git
cd prerender
npm install
node server.js

The default port that goes through Prerender® is 3000. If your Prerender® server is running, you should be able to go here and see the Google homepage. Make sure that your middleware’s prerenderServiceUrl points to http://localhost:3000/. This would be more to test with their own prerender and run it locally. I would almost say this is more of a before you install your middleware test to see how it would look locally. This won’t show up on their site as a prerender hit.

To see a Prerendered page as a web crawler from Google would see it, set your User Agent to in your browser to Googlebot and visit your URL. You can also open your console and run this on a command line: This is to see it working against our service. You should put in here to check the crawl stats on their prerender.io account and they will see requests being sent to their account.

curl -A Googlebot https://www.yourdomain.com/

If the middleware is set up properly, you should see your page returned. You can change your chrome user agent https://developer.chrome.com/docs/devtools/device-mode/override-user-agent/ and see as Googlebot would here as well.

If you’re running into problems, you can identify and troubleshoot them here.

Configure

Installing Prerender and getting it to work properly is one thing. Making it work to suit your specific needs as a website owner and a business is another.

There may be any number of things you need to do after you install Prerender. You may need to make sure it’s only rendering the pages that are having indexing issues.

Here are some best practices to keep in mind:

Indicate When You Want Your Pages to Be Cached

Prerender does its best to identify when a page is done sending HTTP requests, but it’s a good idea to give it some direction so it can do so more efficiently. Put this HTML into your <head> section as near the top of your document as you can:

<script> window.prerenderReady = false; </script>

Then, execute this function after your AJAX calls:

window.prerenderReady = true;

Some more best practices for configuring Prerender® can be found here.

Configure Your Cache 

A problem that Prerender® users encounter sometimes is having the wrong pages cached, or having pages cached unintentionally. You only have so much control over which pages get cached, but if this happens, you can use your middleware to whitelist or blacklist pages.

1. Use the URL parameters section of Prerender.io to ignore specific query parameters that will tell Prerender® to ignore certain URLs. Prerender® can then identify the canonical URL to be cached and then ignore the identical ones
2. Modify your middleware to only send the URLs you indicate to Prerender®. 
3. Set Prerender to return non-200 status codes for pages that you don’t want to be cached.

More information on how to limit your cache can be found here. You can also use the Prerender® recache API to cache a URL that has been cached previously. This can be used to recache some of your pages that change a little more often.https://docs.prerender.io/docs/26-cached-and-recached-pages

Prerender Works With Whatever Techstack You Have

The installation procedures we outlined cover some of the most popular frameworks and languages our customers use, but Prerender is compatible with a wide-range of enterprise languages. You can find a complete list of Prerender-compatible middleware on this page. If you still need some help, you can read more about how to get started with Prerender® here.