es6-crawler-detect
Advanced tools
Readme
The new version now do include a middleware for expressjs, please read the documentation for the latest updates.
This Library is an ES6 version of the original PHP class @CrawlerDetect, it helps you detect bots/crawlers and spiders only by scanning the user-agent string or from the global request.headers.
npm install es6-crawler-detect
'use strict';
const express = require('express')
const {Crawler, middleware} = require('es6-crawler-detect/src')
const app = express()
app.get('your/route', function async (request, response) {
// create a new Crawler instance
var CrawlerDetector = new Crawler(request)
// check the current visitor's useragent
if ( CrawlerDetector.isCrawler() )
{
// true if crawler user agent detected
}
// or check a user agent string
if ( CrawlerDetector.isCrawler('Mozilla/5.0 (compatible; Sosospider/2.0; +http://help.soso.com/webspider.htm)') )
{
// true if crawler user agent detected
}
// Output the name of the bot that matched (if any)
response.send(CrawlerDetector.getMatches())
})
/**
* Or by using the middleware
*/
app.use(middleware)
app.get('/crawler', function async (request, response) {
// or check a user agent string
request.Crawler.isCrawler('Mozilla/5.0 (compatible; Sosospider/2.0; +http://help.soso.com/webspider.htm)')
// Output the name of the bot that matched (if any)
response.send(request.Crawler.getMatches())
})
<head>
<!-- Metas -->
<meta name="title" content="ES6-CrawlerDetect">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
<meta name="description" content="">
<meta name="keywords" content="">
<title>CrawlerDetect - the web crawler detection library</title>
<!-- Scripts -->
<script type="text/javascript" src="./your/path/to/main.bundle.js"></script>
</head>
// create a new Crawler instance
var CrawlerDetector = new Crawler.Crawler();
var userAgentString = navigator.userAgent;
// check the current visitor's useragent
if ( CrawlerDetector.isCrawler(userAgentString) )
{
// true if crawler user agent detected
}
// Output the name of the bot that matched (if any)
console.debug(CrawlerDetector.getMatches());
If you find a bot/spider/crawler user agent that CrawlerDetect fails to detect, please submit a pull request with the regex pattern added to the data array in ./crawler/crawlers.js.
FAQs
This is an ES6 adaptation of the original PHP library CrawlerDetect, this library will help you detect bots/crawlers/spiders vie the useragent.
We found that es6-crawler-detect demonstrated a not healthy version release cadence and project activity because the last version was released a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Application Security
New SEC disclosure rules aim to enforce timely cyber incident reporting, but fear of job loss and inadequate resources lead to significant underreporting.
Security News
The Python Software Foundation has secured a 5-year sponsorship from Fastly that supports PSF's activities and events, most notably the security and reliability of the Python Package Index (PyPI).
Security News
LDAPjs, an LDAP Client and Server API for Node.js, was decommissioned after its maintainer received an abusive email from a user, raising concerns about this form of abuse as a potential attack vector.