nodejs-web-scraper

Nodejs-web-scraper is a simple, yet powerful and feature-rich tool, for Node programmers who want to quickly setup a complex scraping job of server-side rendered web sites.

Basic example

Nodejs-web-scraper has a semantic API:

const { Scraper, Root, DownloadContent, Inquiry, OpenLinks, CollectContent } =  require('nodejs-web-scraper');    

var config = {
        baseSiteUrl: `https://www.nytimes.com/`,
		startUrl: `https://www.nytimes.com/`,
		concurrency: 10,
     	fileFlag: 'w',
    	maxRetries: 3,
	    cloneImages: true,
	    delay: 50,	 
	    filePath: './images/',
	    logPath: './logs/'
}  

var scraper =  new  Scraper(config);//Create a new Scraper instance, and pass config to it.

var root =  new  Root();//The root object is responsible for fetching the startUrl, and starting the scraping process.  

const category =  new  OpenLinks('.css-1wjnrbv', { name: 'category' });//Opens each category page.

const article =  new  OpenLinks('article a', { name: 'article' });//Opens each article page

const h1 =  new  CollectContent('h1', { name: 'h1' });//"Collects" the text from each H1 element.

const image =  new  DownloadContent('img', { name: 'image' });//Downloads every image from a given page.  

root.addOperation(category);//We add child objects to their parents
	category.addOperation(article);
		article.addOperation(image);
		article.addOperation(h1);

This basically means: "go to www.nytimes.com . Open every category. Then open every article in each category page. Then collect the h1 tags in each article, and download all images on that page".