spider-detector
A tiny node module to detect spiders/crawlers quickly and comes with optional middleware for ExpressJS
It might be useful when you have a single page app but want to deliver static pages for spiders.
Install
npm install spider-detector
Direct Example
var detector = require('spider-detector')
detector.isSpider('baiduspider')
ExpressJS example
var detector = require('spider-detector'),
express = require('express'),
app = express()
app.use(detector.middleware())
app.get('/*', function(req, res) {
if (req.isSpider()) {
} else {
}
})
Why? There are already modules out there
Well, I wanted one which does not use readFileSync
and comes with optional middleware. Furthermore some hackers do not classify Googlebot as a spider anymore which poses a problem sometimes, see next question.
What about Googlebot?
Yep, Googlebot is able to deal with single page apps but this feature is pretty unstable. Especially under AngularJS when hash fragments are disabled with $locationProvider.html5Mode(true)
. That's why - against all odds - I have classified googlebot
as a spider in this module.