@vostersc/crawler

Crawler

This package is a wrapper around Puppeteer. The package provides faster out of the box use by taking care of things like proxy urls, ips, user agents, and others.

Quick Start

1. In terminal run npm i @vostersc/crawler
2. Import const crawler = require('@vostersc/crawler');
3. Write own selector function or use example selector. Console log the selector function for help writing your own. console.log(crawler.exampleSelectorFunction);
4. Run crawler. const result = await crawler.getData([{url: url1, selector: crawler.exampleSelectorFunction}]);
5. Your whole file should look something like this:

const crawler = require('@vostersc/crawler');

(async function(){
  const result = await crawler.getData([{url: 'https://reddit.com', selector: crawler.exampleSelectorFunction}]);
  console.log(result);
})()

Quick Start: Actions

1. Follow the steps above but your final file should look something like this:

const crawler = require('@vostersc/crawler');

(async function(){
    const urls = [
        'http://shop.harmonsgrocery.com/search?search_term=black pepper',
        'http://shop.harmonsgrocery.com/search?search_term=banana',
        'http://shop.harmonsgrocery.com/search?search_term=1/2 cup flour'
    ];
    const qty = [7,1, 0];
    console.log(await doAction(urls, exampleActionFunction, qty));
})()