spiderable-middleware
Advanced tools
Perfect SEO for JavaScript websites. Pre-rendering — it's just like SSR with simple integration and no coding
Google, Facebook, Twitter, Yahoo, and Bing and all other crawlers and search engines are constantly trying to view your website. If your website is built on top of the JavaScript framework like, but not limited to - Angular, Backbone, Ember, Meteor, React, MEAN most of the front-end solutions returns basic HTML-markup and script-tags to crawlers, but not content of your page. The mission of spiderable-middleware and ostr.io are to boost your SEO experience without a headache.
styled-components;Content-Security-Policy and other "complicated" front-end security;This package acts as middleware and intercepts requests to your Node.js application from web crawlers. All requests proxy passed to the Prerendering Service, which returns static, rendered HTML.
This is SERVER only package. For NPM make sure you're importing library only in Node.js. For Meteor.js please make sure library imported and executed only on SERVER.
We made this package with developers in mind. It's well written in a very simple way, hackable, and easily tunable to meet your needs, can be used to turn dynamic pages into rendered, cached, and lightweight static pages, just set botsUA to ['.*']. This is the best option to offload servers unless a website gets updated more than once in 4 hours.
Location and others. Read how to return genuine status code and handle JS-redirects.This middleware was tested and works like a charm with:
All other frameworks which follow Node's middleware convention - will work too.
This package was originally developed for ostr.io service. But it's not limited to, and can proxy-pass requests to any other rendering-endpoint.
Install spiderable-middleware package from NPM for Node.js usage, alternatively see Meteor.js specific setup documentation
npm install spiderable-middleware --save
Get ready in a few lines of code
See all examples.
First, add fragment meta-tag to your HTML template:
<html>
<head>
<meta name="fragment" content="!">
<!-- ... -->
</head>
<body>
<!-- ... -->
</body>
</html>
// Make sure this code isn't exported to the Browser bundle
// and executed only on SERVER (Node.js)
const express = require('express');
const Spiderable = require('spiderable-middleware');
const spiderable = new Spiderable({
rootURL: 'http://example.com',
auth: 'APIUser:APIPass'
});
const app = express();
app.use(spiderable.handler).get('/', (req, res) => {
res.send('Hello World');
});
app.listen(3000);
We provide various options for serviceURL as "Rendering Endpoints", each endpoint has its own features to fit every project needs.
To pass expected response code from front-end JavaScript framework to browser/crawlers, you need to create specially formatted HTML-comment. This comment can be placed in any part of HTML-page. head or body tag is the best place for it.
html:
<!-- response:status-code=404 -->
jade:
// response:status-code=404
This package support any standard and custom status codes:
201 - <!-- response:status-code=201 -->401 - <!-- response:status-code=401 -->403 - <!-- response:status-code=403 -->500 - <!-- response:status-code=500 -->514 - <!-- response:status-code=514 --> (non-standard)To speed-up rendering, you should tell to the Spiderable engine when your page is ready. Set window.IS_RENDERED to false, and once your page is ready set this variable to true. Example:
<html>
<head>
<meta name="fragment" content="!">
<script>
window.IS_RENDERED = false;
</script>
</head>
<body>
<!-- ... -->
<script type="text/javascript">
//Somewhere deep in your app-code:
window.IS_RENDERED = true;
</script>
</body>
</html>
Pre-rendering engine will set window.IS_PRERENDERING global variable to true. Detecting requests from pre-rendering engine are as easy as:
if (window.IS_PRERENDERING) {
// This request is coming from Pre-rendering engine
}
Note: window.IS_PRERENDERING can be undefined on initial page load, and may change during runtime. That's why we recommend to pre-define a setter for IS_PRERENDERING:
let isPrerendering = false;
Object.defineProperty(window, 'IS_PRERENDERING', {
set(val) {
isPrerendering = val;
if (isPrerendering === true) {
// This request is coming from Pre-rendering engine
}
},
get() {
return isPrerendering;
}
});
If you need to redirect browser/crawler inside your application, while a page is loading (imitate navigation), you're free to use any of classic JS-redirects as well as your framework's navigation, or History.pushState()
window.location.href = 'http://example.com/another/page';
window.location.replace('http://example.com/another/page');
Router.go('/another/page'); // framework's navigation !pseudo code
Note: Only 4 redirects are allowed during one request after 4 redirects session will be terminated.
new Spiderable([opts])opts {Object} - Configuration optionsopts.serviceURL {String} - Valid URL to Spiderable endpoint (local or foreign). Default: https://render.ostr.io. Can be set via environment variables: SPIDERABLE_SERVICE_URL or PRERENDER_SERVICE_URLopts.rootURL {String} - Valid root URL of your website. Can be set via an environment variable: ROOT_URLopts.auth {String} - [Optional] Auth string in next format: user:pass. Can be set via an environment variables: SPIDERABLE_SERVICE_AUTH or PRERENDER_SERVICE_AUTH. Default nullopts.sanitizeUrls {Boolean} - [Optional] Sanitize URLs in order to "fix" badly composed URLs. Default falseopts.botsUA {[String]} - [Optional] An array of strings (case insensitive) with additional User-Agent names of crawlers you would like to intercept. See default bot's names. Set to ['.*'] to match all browsers and robots, to serve static pages to all users/visitorsopts.ignoredHeaders {[String]} - [Optional] An array of strings (case insensitive) with HTTP header names to exclude from response. See default list of ignored headers. Set to ['.*'] to ignore all headersopts.ignore {[String]} - [Optional] An array of strings (case sensitive) with ignored routes. Note: it's based on first match, so route /users will cause ignoring of /part/users/part, /users/_id and /list/of/users, but not /user/_id or /list/of/blocked-users. Default nullopts.only {[String|RegExp]} - [Optional] An array of strings (case sensitive) or regular expressions (could be mixed). Define exclusive route rules for pre-rendering. Could be used with opts.onlyRE rules. Note: To define "safe" rules as {RegExp} it should start with ^ and end with $ symbols, examples: [/^\/articles\/?$/, /^\/article\/[A-z0-9]{16}\/?$/]opts.onlyRE {RegExp} - [Optional] Regular Expression with exclusive route rules for pre-rendering. Could be used with opts.only rulesopts.timeout {Number} - [Optional] Number, proxy-request timeout to rendering endpoint in milliseconds. Default: 180000opts.requestOptions {Object} - [Optional] Options for request module (like: timeout, lookup, insecureHTTPParser), for all available options see http API docsopts.debug {Boolean} - [Optional] Enable debug and extra logging, default: falseNote: Setting .onlyRE and/or .only rules are highly recommended. Otherwise, all routes, including randomly generated by bots will be subject of Pre-rendering and may cause unexpectedly higher usage.
// CommonJS
// const Spiderable = require('spiderable-middleware');
// ES6 import
// import Spiderable from 'spiderable-middleware';
const spiderable = new Spiderable({
rootURL: 'http://example.com',
auth: 'APIUser:APIPass'
});
// More complex setup (recommended):
const spiderable = new Spiderable({
rootURL: 'http://example.com',
serviceURL: 'https://render.ostr.io',
auth: 'APIUser:APIPass',
only: [
/\/?/, // Root of the website
/^\/posts\/?$/, // "/posts" path with(out) trailing slash
/^\/post\/[A-z0-9]{16}\/?$/ // "/post/:id" path with(out) trailing slash
],
// [Optionally] force ignore for secret paths:
ignore: [
'/account/', // Ignore all routes under "/account/*" path
'/billing/' // Ignore all routes under "/billing/*" path
]
});
Same configuration can get achieved via setting up environment variables:
ROOT_URL='http://example.com'
SPIDERABLE_SERVICE_URL='https://render.ostr.io'
SPIDERABLE_SERVICE_AUTH='APIUser:APIPass'
alternatively if you're migrating from other pre-rendering service you can keep using existing variables, we support it for compatibility
ROOT_URL='http://example.com'
PRERENDER_SERVICE_URL='https://render.ostr.io'
PRERENDER_SERVICE_AUTH='APIUser:APIPass'
spiderable.handler(req, res, next)Middleware handler. Alias: spiderable.handle.
// Express, Connect:
app.use(spiderable.handler);
// Meteor:
WebApp.connectHandlers.use(spiderable);
//HTTP(s) Server
http.createServer((req, res) => {
spiderable.handler(req, res, () => {
// Callback, triggered if this request
// is not a subject of spiderable pre-rendering
res.writeHead(200, {'Content-Type': 'text/plain; charset=UTF-8'});
res.end('Hello vanilla NodeJS!');
// Or do something else ...
});
}).listen(3000);
To properly serve pages for Accelerated Mobile Pages (AMP) we support following URI schemes:
# Regular URIs:
https://example.com/index.html
https://example.com/articles/article-title.html
https://example.com/articles/article-uniq-id/article-slug
# AMP optimized URIs (prefix):
https://example.com/amp/index.html
https://example.com/amp/articles/article-title.html
https://example.com/amp/articles/article-uniq-id/article-slug
# AMP optimized URIs (extension):
https://example.com/amp/index.amp.html
https://example.com/amp/articles/article-title.amp.html
All URLs with .amp. extension and /amp/ prefix will be optimized for AMP.
https://render.ostr.io - This endpoint has "optimal" settings, and should fit 98% cases. Respects cache headers of both Crawler and your server;https://render-bypass.ostr.io - This endpoint has bypass caching mechanism (almost no cache). Use it if you're experiencing an issue, or during development, to make sure you're not running into the intermediate cache. You're safe to use this endpoint in production, but it may result in higher usage and response time.https://render-cache.ostr.io - This endpoint has the most aggressive caching mechanism. Use it if you're looking for the shortest response time, and don't really care about outdated pages in cache for 6-12 hoursTo change default endpoint, grab integration examples code and replace render.ostr.io, with endpoint of your choice. For NPM/Meteor integration change value of serviceURL option.
Note: Described differences in caching behavior related to intermediate proxy caching, Cache-Control header will be always set to the value defined in "Cache TTL". Cached results at the "Pre-rendering Engine" end can be purged at any time.
To make sure a server can reach our rendering endpoint run cURL command or send request via Node.js to (replace example.com with your domain name) https://test:test@render-bypass.ostr.io/?url=http://example.com.
In this example we're using render-bypass.ostr.io endpoint to avoid any possible cached results, read more about rendering endpoints. As API credentials we're using test:test, this part of URL can be replaced with auth option from Node.js example. Your API credentials and instructions can be found at the very bottom of Pre-rendering Panel of a host, click on "Integration Guide".
# cURL example:
curl -v "https://test:test@render-bypass.ostr.io/?url=http://example.com"
// Node.js example:
const https = require('https');
https.get('https://test:test@render-bypass.ostr.io/?url=http://example.com', (resp) => {
let data = '';
resp.on('data', (chunk) => {
data += chunk.toString('utf8');
});
resp.on('end', () => {
console.log(data);
});
}).on('error', (error) => {
console.error(error);
});
# Install development NPM dependencies:
npm install --save-dev
# Install NPM dependencies:
npm install --save
# Run tests:
ROOT_URL=http://127.0.0.1:3003 npm test
# Run same tests with extra-logging
DEBUG=true ROOT_URL=http://127.0.0.1:3003 npm test
# http://127.0.0.1:3003 can be changed to any local address, PORT is required!
FAQs
Perfect SEO for JavaScript websites. Pre-rendering — it's just like SSR with simple integration and no coding
The npm package spiderable-middleware receives a total of 6 weekly downloads. As such, spiderable-middleware popularity was classified as not popular.
We found that spiderable-middleware demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
In this segment of the Risky Business podcast, Feross Aboukhadijeh and Patrick Gray discuss the challenges of tracking malware discovered in open source softare.
Research
Security News
A threat actor's playbook for exploiting the npm ecosystem was exposed on the dark web, detailing how to build a blockchain-powered botnet.
Security News
NVD’s backlog surpasses 20,000 CVEs as analysis slows and NIST announces new system updates to address ongoing delays.