punch-scraper

punch-scraper

Config

• proxyManagerConfig - punch proxy manager config
• maxTry - How many time scrapper will try to fetch the link before error
• strategy - scraper strategies
• name - strategy name valid values: CASPERJS, HTTP, PHANTOMJS
• proxy - proxy ip
• lambda - for CASPERJS or PHANTOMJS only

•  aws_key - aws key
  

•  aws_secret - aws secret key
  

•  region - aws region
  

•  lambda_name - aws labmda function name
  

• eval - code that should be evaled for CASPERJS or PHANTOMJS only
• services
• include - array of proxy services to use
• exclude - array of proxy services to not use
• valid valuesGIMMI_PROXY, HIDE_MY_ASS, IN_CLOCK, PROXY_SERVER_LIST, UK_PROXY, US_PROXY

Method

• scrape - scrape urls
• start - start the scraper manager
• stop - stop the scraper manager

Usage

'use strict';

const ScrapeManager = require('./scraper-manager/');
const scrapeManager = new ScrapeManager();
const config = {
    eval: "response.write(page.content);response.close();",
    strategy: {
        name: 'phantomjs',
        lambda: {
            aws_key: 'XXX-XXX-XXX',
            aws_secret: 'XXX-XXX-XXX',
            lambda_name: 'node-phantomjs-aws-lambda-server-development',
            region: 'us-west-2'
        }
    }
};

let links = [
  'http://www.google.com/',
  'http://www.google.com/'
];


scrapeManager.start()
.then(() => scrapeManager.scrape(links, config))
.then((results) => {
    console.log(results);
    console.log('done');
    scrapeManager.stop();
});