@vostersc/crawler

Crawler

This package is a wrapper around Puppeteer. The package provides faster out of the box use by taking care of things like proxy urls, ips, user agents, and others.

Quick Start

const C = new Crawler();

const config = [
    {selector: C.exampleActionFunction, urls: ['http://shop.harmonsgrocery.com/search?search_term=peach']},
    {selector: C.exampleActionFunction, urls: ['http://shop.harmonsgrocery.com/search?search_term=avocado']},
    {selector: C.exampleActionFunction, urls: ['http://shop.harmonsgrocery.com/search?search_term=steak']}
];
const anon = false;
const logger = false;
const keepOpen = true;

C.performAction(config, anon, logger, keepOpen); //ignore the await

setTimeout(() => {
    console.log('status', C.getStatus());
}, 30000);