es6-crawler-detect

This is an ES6 adaptation of the original PHP library CrawlerDetect, this library will help you detect bots/crawlers/spiders vie the useragent.

3.3.0

latest

source

npm

Version published: 2 years ago

Weekly downloads: 12K; decreased by-28.59%

Maintainers: 1

Install size: 25.5 MB

Created: 7 years ago

Weekly downloads

Readme

Source

Crawler Detect

crawlerdetect.io

About

This Library is an ES6 version of the original PHP class @CrawlerDetect, it helps you detect bots/crawlers and spiders only by scanning the user-agent string or from the global request.headers.

Installation

npm install es6-crawler-detect

Usage

ECMAScript 6 (ES6)

'use strict';

const express = require('express')
const { Crawler, middleware } = require('es6-crawler-detect')

const app = express()

app.get('your/route', function async (request, response) {
  // create a new Crawler instance
  var CrawlerDetector = new Crawler(request)
  
  // check the current visitor's useragent
  if ( CrawlerDetector.isCrawler() )
  {
    // true if crawler user agent detected
  }
  
  // or check a user agent string
  if ( CrawlerDetector.isCrawler('Mozilla/5.0 (compatible; Sosospider/2.0; +http://help.soso.com/webspider.htm)') )
  {
    // true if crawler user agent detected
  }
  
  // Output the name of the bot that matched (if any)
  response.send(CrawlerDetector.getMatches())
})

/**
 * Or by using the middleware
*/
app.use(middleware((request, reponse) => {
  // do something here
  // e.g. console.log(request.body)
  // e.g. return response.status(403).send('Forbidden')
}))

app.get('/crawler', function async (request, response) {
  // or check a user agent string
  request.Crawler.isCrawler('Mozilla/5.0 (compatible; Sosospider/2.0; +http://help.soso.com/webspider.htm)')
  
  // Output the name of the bot that matched (if any)
  response.send(request.Crawler.getMatches())
})

ECMAScript 5 (ES5)

<head>
  <!-- Metas -->
  <meta name="title" content="ES6-CrawlerDetect">
  <meta http-equiv="X-UA-Compatible" content="IE=edge">
  <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
  <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
  <meta name="description" content="">
  <meta name="keywords" content="">
  <title>CrawlerDetect - the web crawler detection library</title>
  <!-- Scripts -->
  <script type="text/javascript" src="./your/path/to/main.bundle.js"></script>
</head>

  // create a new Crawler instance
  var CrawlerDetector = new Crawler.Crawler();
  var userAgentString = navigator.userAgent;
  
  // check the current visitor's useragent
  if ( CrawlerDetector.isCrawler(userAgentString) )
  {
    // true if crawler user agent detected
  }
  
  // Output the name of the bot that matched (if any)
  console.debug(CrawlerDetector.getMatches());

Contributing

If you find a bot/spider/crawler user agent that CrawlerDetect fails to detect, please submit a pull request with the regex pattern added to the data array in ./crawler/crawlers.js.

Keywords

FAQs

What is es6-crawler-detect?

Is es6-crawler-detect popular?

Is es6-crawler-detect well maintained?

Last updated on 06 Jan 2022

Did you know?

Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.

Install