Spiderable middleware
Spiderable Middleware is a lightweight Node.js package designed for modern JavaScript web apps — including those built with React, Preact, Vue, Svelte, Angular, Ember, Backbone, Meteor, and others. It ensures that search engine crawlers and social media bots receive fully-rendered HTML, not just an empty skeleton. This bridges the gap between client-rendered apps and the indexing and preview requirements of platforms like Google, Facebook, and Twitter.
When paired with the global CDN and smart caching layer from ostr.io
, spiderable-middleware
significantly improves SEO scores, link previews, and performance metrics like TTFB and Lighthouse scores — all without requiring changes to existing codebase. It intelligently reroutes bot traffic to ostr.io
rendering endpoints, minimizing server load, reducing database queries, and lowering infrastructure costs.
Why Pre-render?
- 🕸 Execute JavaScript, — get rendered HTML page and its content;
- 🏎️ Improve delivery for dynamic and static pages via our advanced CDN and caching;
- 🏃♂️ Boost response rate and decrease response time with caching;
- 🚀 Optimized HTML markup for the best SEO score;
- 🎛️ Improve TTFB, LCP, INP, CLS, and other LightHouse metrics positively enhancing overall SEO score;
- 🖥 Supports PWAs and SPAs;
- 📱 Supports mobile-like crawlers;
- 💅 Supports
styled-components
;
- ⚡️ Supports AMP (Accelerated Mobile Pages);
- 🤓 Works with
Content-Security-Policy
and other complex front-end security rules;
- 📦 This package shipped with types and TS examples;
- ❤️ Search engines and social network crawlers love straightforward and pre-rendered pages;
- 📱 Consistent link previews in messaging apps, like iMessage, Messages, Facebook, Slack, Telegram, WhatsApp, Viber, VK, Twitter, and other apps;
- 💻 Image, title, and description previews for links posted at social networks, like Facebook, X/Twitter, Instagram, and other social networks.
ToC
About Package
This package works as Express-style middleware, intercepting requests from crawlers and social media bots to your Node.js application. It seamlessly proxies those requests to a pre-rendering service, which returns fully-rendered static HTML — optimized for indexing and rich previews.
Built with developers in mind, spiderable-middleware
is lightweight, well-structured, and easy to customize. Whether you're scaling a production app or prototyping, it's designed to be hackable and flexible enough to fit any project. By offloading bot traffic to the pre-rendering engine, it helps reduce backend load and improve server performance effortlessly.
[!NOTE]
This package proxies real HTTP headers and response code, to reduce overwhelming requests, try to avoid HTTP-redirect headers, like Location
. Read how to return genuine status code and handle JS-redirects
[!IMPORTANT]
This is server only package. This package should be imported/initialized only within server codebase
This middleware was tested and works like a charm with:
All other frameworks that follows Node/Express middleware convention - will work too.
[!TIP]
This package was originally developed for ostr.io service. But it's not limited to, and can proxy-pass requests to any other rendering-endpoint.
Installation
Install spiderable-middleware
package from NPM:
npm install spiderable-middleware --save
yarn add spiderable-middleware
Usage
Setup pre-rendering middleware in few lines of code
Basic usage
Start with adding fragment
meta-tag to HTML template, head, or page:
[!TIP]
See usage examples in .js
and .ts
for quick copy-paste experience
<html>
<head>
<meta name="fragment" content="!">
</head>
<body>
</body>
</html>
Import or require spiderable-middleware
package:
import Spiderable from 'spiderable-middleware';
const Spiderable = require('spiderable-middleware');
Register middleware handle:
import express from 'express';
import Spiderable from 'spiderable-middleware';
const spiderable = new Spiderable({
rootURL: 'http://example.com',
auth: 'test:test',
});
const app = express();
app.use(spiderable.handle).get('/', (req, res) => {
res.send('Hello World');
});
app.listen(3000);
[!TIP]
We provide various options for serviceURL
as "Rendering Endpoints", each endpoint has its own features to fit different project needs
Return genuine status code
To pass expected status code of a response from front-end JavaScript framework to browser/crawlers use specially formatted HTML-comment. This comment can be placed in any part of HTML-page. head
or body
tag is the best place for it.
Format
html:
jade:
// response:status-code=404
This package support any standard and custom status codes:
201
- <!-- response:status-code=201 -->
401
- <!-- response:status-code=401 -->
403
- <!-- response:status-code=403 -->
500
- <!-- response:status-code=500 -->
514
- <!-- response:status-code=514 -->
(non-standard)
Speed-up rendering
To speed-up rendering, JS-runtime should tell to the Spiderable engine when the page is ready. Set window.IS_RENDERED
to false
, and once the page is ready set this variable to true
. Example:
<html>
<head>
<meta name="fragment" content="!">
<script>
window.IS_RENDERED = false;
</script>
</head>
<body>
<script type="text/javascript">
window.IS_RENDERED = true;
</script>
</body>
</html>
Detect request from Pre-rendering engine during runtime
Pre-rendering engine will set window.IS_PRERENDERING
global variable to true
. Detecting requests from pre-rendering engine are as easy as:
if (window.IS_PRERENDERING) {
}
[!NOTE]
window.IS_PRERENDERING
can be undefined
on initial page load, and may change during runtime. That's why we recommend to pre-define a setter for IS_PRERENDERING
:
let isPrerendering = false;
Object.defineProperty(window, 'IS_PRERENDERING', {
set(val) {
isPrerendering = val;
if (isPrerendering === true) {
}
},
get() {
return isPrerendering;
}
});
Detect type of the Pre-rendering engine
Like browsers, — crawlers and bots may request page as "mobile" (small screen touch-devices) or as "desktop" (large screens without touch-events) the pre-rendering engine supports these two types. For cases when content needs to get optimized for different screens pre-rendering engine will set window.IS_PRERENDERING_TYPE
global variable to desktop
or mobile
if (window.IS_PRERENDERING_TYPE === 'mobile') {
} else if (window.IS_PRERENDERING_TYPE === 'desktop') {
} else {
}
JavaScript redirects
Redirect browser/crawler inside application when needed while a page is loading (imitate navigation), use any of classic JS-redirects can be used, including framework's navigation, or History.pushState()
window.location.href = 'http://example.com/another/page';
window.location.replace('http://example.com/another/page');
Router.go('/another/page');
[!IMPORTANT]
Only 4 redirects are allowed during one request after 4 redirects session will be terminated.
API
Create new instance and pass middleware to server's routes chain;
Constructor
new Spiderable(opts?: SpiderableOptions);
opts
{SpiderableOptions?} - [Optional] Configuration options
opts.serviceURL
{string} - Valid URL to Spiderable endpoint (local or foreign). Default: https://render.ostr.io
. Can be set via environment variables: SPIDERABLE_SERVICE_URL
or PRERENDER_SERVICE_URL
opts.rootURL
{string} - Valid root URL of a website. Can be set via an environment variable: ROOT_URL
opts.auth
{string} - Auth string in next format: user:pass
. Can be set via an environment variables: SPIDERABLE_SERVICE_AUTH
or PRERENDER_SERVICE_AUTH
. Default null
opts.sanitizeUrls
{boolean} - Sanitize URLs in order to "fix" badly composed URLs. Default false
opts.botsUA
{string[]} - An array of strings (case insensitive) with additional User-Agent names of crawlers that needs to get intercepted. See default bot's names. Set to ['.*']
to match all browsers and robots, to serve static pages to all users/visitors
opts.ignoredHeaders
{string[]} - An array of strings (case insensitive) with HTTP header names to exclude from response. See default list of ignored headers. Set to ['.*']
to ignore all headers
opts.ignore
{string[]} - An array of strings (case sensitive) with ignored routes. Note: it's based on first match, so route /users
will cause ignoring of /part/users/part
, /users/_id
and /list/of/users
, but not /user/_id
or /list/of/blocked-users
. Default null
opts.only
{(String|RegExp)[]} - An array of strings (case sensitive) or regular expressions (could be mixed). Define exclusive route rules for pre-rendering. Could be used with opts.onlyRE
rules. Note: To define "safe" rules as {RegExp} it should start with ^
and end with $
symbols, examples: [/^\/articles\/?$/, /^\/article\/[A-z0-9]{16}\/?$/]
opts.onlyRE
{RegExp} - Regular Expression with exclusive route rules for pre-rendering. Could be used with opts.only
rules
opts.timeout
{number} - Number, proxy-request timeout to rendering endpoint in milliseconds. Default: 180000
opts.requestOptions
{RequestOptions} - Options for request module (like: timeout
, lookup
, insecureHTTPParser
), for all available options see http
API docs
opts.debug
{boolean} - [Optional] Enable debug and extra logging, default: false
[!IMPORTANT]
Setting .onlyRE
and/or .only
rules are highly recommended. Otherwise, all routes, including randomly generated by bots will be subject of Pre-rendering and may cause unexpectedly higher usage.
const spiderable = new Spiderable({
rootURL: 'http://example.com',
auth: 'test:test'
});
const spiderable = new Spiderable({
rootURL: 'http://example.com',
serviceURL: 'https://render.ostr.io',
auth: 'test:test',
only: [
/\/?/,
/^\/posts\/?$/,
/^\/post\/[A-z0-9]{16}\/?$/
],
ignore: [
'/account/',
'/billing/'
]
});
Configuration via env.vars
Same configuration can get achieved via setting up environment variables:
ROOT_URL='http://example.com'
SPIDERABLE_SERVICE_URL='https://render.ostr.io'
SPIDERABLE_SERVICE_AUTH='APIUser:APIPass'
alternatively, when migrating from other pre-rendering service — keep using existing variables, we support the next ones for compatibility:
ROOT_URL='http://example.com'
PRERENDER_SERVICE_URL='https://render.ostr.io'
PRERENDER_SERVICE_AUTH='APIUser:APIPass'
handle
Middleware handle
const spiderable = new Spiderable();
spiderable.handle(req: IncomingMessage, res: ServerResponse, next: NextFunction): void;
spiderable.handler(req: IncomingMessage, res: ServerResponse, next: NextFunction): boolean;
Example using connect
and express
package:
import { createServer } from 'node:http';
import Spiderable from 'spiderable-middleware';
const app = express();
const spiderable = new Spiderable();
app.use(spiderable.handle).use((_req, res) => {
res.end('Hello from Connect!\n');
});
createServer(app).listen(3000;
Example using node.js http
server:
import { createServer } from 'node:http';
import Spiderable from 'spiderable-middleware';
http.createServer((req, res) => {
spiderable.handle(req, res, () => {
res.writeHead(200, {'Content-Type': 'text/plain; charset=UTF-8'});
res.end('Hello vanilla NodeJS!');
});
}).listen(3000);
Types
Import types right from NPM package
import Spiderable from 'spiderable-middleware';
import type { SpiderableOptions, NextFunction } from 'spiderable-middleware';
const options: SpiderableOptions = {
rootURL: 'http://example.com',
auth: 'test:test',
debug: false,
};
expectType<SpiderableOptions>(options);
const spiderable = new Spiderable(options);
expectType<Spiderable>(spiderable);
const next: NextFunction = (_err?: unknown): void => {};
expectType<void>(spiderable.handle(req, res, next));
AMP Support
To properly serve pages for Accelerated Mobile Pages (AMP) we support following URI schemes:
https://example.com/index.html
https://example.com/articles/article-title.html
https://example.com/articles/article-uniq-id/article-slug
https://example.com/amp/index.html
https://example.com/amp/articles/article-title.html
https://example.com/amp/articles/article-uniq-id/article-slug
https://example.com/amp/index.amp.html
https://example.com/amp/articles/article-title.amp.html
[!IMPORTANT]
All URLs with .amp.
extension and /amp/
prefix will be optimized for AMP.
Rendering Endpoints
- render (default) -
https://render.ostr.io
- This endpoint has "optimal" settings, and should fit 98% cases. This endpoint respects cache headers of Crawler and origin server
- render-bypass (devel/debug) -
https://render-bypass.ostr.io
- This endpoint will bypass caching mechanisms. Use it when experiencing an issue, or during development, to make sure responses are not cached. It's safe to use this endpoint in production, but it may result in higher usage and response time
- render-cache (under attack) -
https://render-cache.ostr.io
- This endpoint has the most aggressive caching mechanism. Use it to achieve the shortest response time, and when outdated pages (for 6-12 hours) are acceptable
To change default endpoint, grab integration examples code and replace render.ostr.io
, with endpoint from the list above. For NPM integration change value of serviceURL
option.
Note: Described differences in caching behavior related to intermediate proxy caching, Cache-Control
header will be always set to the value defined in "Cache TTL". Cached results at the "Pre-rendering Engine" end can be purged at any time.
Convert dynamic website to static
spiderable-middleware
package can get used to convert dynamic websites to rendered, cached, and lightweight static pages. Simply set botsUA
to ['.*']
to achieve this behavior
import Spiderable from 'spiderable-middleware';
const spiderable = new Spiderable({
botsUA: ['.*']
});
Debugging
Pass { debug: true }
or set DEBUG=true
environment variable to enable debugging mode.
[!TIP]
To make sure a server can reach a rendering endpoint run cURL
command or send request via Node.js to (replace example.com with your domain name):
# cURL example:
curl -v "https://test:test@render-bypass.ostr.io/?url=http://example.com"
In this example we're using render-bypass.ostr.io
endpoint to avoid any possible cached results, read more about rendering endpoints. As API credentials we're using test:test
, this part of URL can be replaced with auth
option from Node.js example.
[!TIP]
The API credentials and instructions can be found at the very bottom of Pre-rendering Panel, — click on the name of your website, then on Integration Guide at the bottom of the page
const https = require('https');
https.get('https://test:test@render-bypass.ostr.io/?url=http://example.com', (resp) => {
let data = '';
resp.on('data', (chunk) => {
data += chunk.toString('utf8');
});
resp.on('end', () => {
console.log(data);
});
}).on('error', (error) => {
console.error(error);
});
Running Tests
- Clone this package
- In Terminal (Console) go to directory where package was cloned
- Then run:
Node.js/Mocha
npm install --save-dev
npm install --save
npm link
npm link spiderable-middleware
ROOT_URL=http://127.0.0.1:3003 npm test
DEBUG=true ROOT_URL=http://127.0.0.1:3003 npm test