crawler-request
HTTP request module customized for crawlers.
Installation
npm install crawler-request
Usage
Simple Request
const crawler = require('crawler-request');
crawler("https://stackoverflow.com/").then(function(response){
console.log(response.text.lenght);
});
PDF Parse
const crawler = require('crawler-request');
crawler("http://careers.stackoverflow.com/stack_overflow_careers.pdf").then(function(response){
console.log(response.text.lenght);
});
Extend
const crawler = require('crawler-request');
function response_text_size(response){
response["size"] = response.text.length;
return response;
}
crawler("http://journals.tubitak.gov.tr/medical/issues/sag-09-39-3/sag-39-3-4-0902-21.pdf",response_text_size).then(function(response){
console.log(response.size);
});
Test
mocha
or npm test
check test folder and QUICKSTART.js for extra usages.
Support
I use this package actively myself, so it has my top priority.
Submitting an Issue
If you find a bug or a mistake, you can help by submitting an issue to GitLab Repository
Creating a Merge Request
GitLab calls it merge request instead of pull request.
Contact me on WhatsApp
For discussing about the package.
License
MIT licensed and all it's dependencies are MIT or BSD licensed.