Security News
Cloudflare Adds Security.txt Setup Wizard
Cloudflare has launched a setup wizard allowing users to easily create and manage a security.txt file for vulnerability disclosure on their websites.
html-scrapper
Advanced tools
A general purpose scrapper written using cheerio, request, async etc. It can scrap data from html pages based on some schema defnition. A schema denition includes a css selector and a function that accepts this element and returns the data. It is simple
A general purpose web scrapper written in server side JavaScript. It is written for ease of use.
#Example Example of scrapping github's explore page.
var scrapper = require('../');
var Source = scrapper.Source;
var Extractor = scrapper.Extractor;
var Fn = scrapper.Fn;
var github = new Source('get', 'https://github.com/explore' );
var dataSchema = {
trendingRepos:[ {
$rule: '#explore-trending .explore-collection > ul > li',
name: '.repo-name',
forks: ':nth-child(1)',
stars: {
$rule: ':nth-child(2)',
$fn: Fn.asInt
}
}]
};
var extractor = new Extractor( dataSchema );
github.read(function(err, res ){
var data = extractor.extract( res.body );
console.log( data );
});
/*
Returns
{ trendingRepos:
[ { name: 'calmh/syncthing', forks: '77', stars: 2019 },
{ name: 'quilljs/quill', forks: '47', stars: 1312 },
{ name: 'filamentgroup/tablesaw', forks: '31', stars: 1128 },
{ name: 'atom/atom', forks: '142', stars: 1035 },
{ name: 'dennis714/RE-for-beginners', forks: '67', stars: 1072 },
{ name: 'mdo/wtf-forms', forks: '43', stars: 912 } ] }
*/
#Usage
Fn contains some usefull data extraction functions that can be used as $fn.
Available functions:
A simple Browser class implementation. It uses request
module for http requests and stores session data in its instance.
Only get
method is implemented right now.
A simple web Crawler class. It uses the following libraries
It need following data to be passed to its constructor.
loadPageList
: A function with signature function( pageLoaderData, cb )
. It is used to collect urls of pages that need to be scraped.
bundle.$endReached
is set true
, then it will stop furthon invocation of loadPageList function.function(err, [ pageData, ... ])
scrapePage
: A function function( pageData, cb)
.
bs
: A BufferedSink instance used to write the data to output medium. See examples/blogspot for a simple implementation that appends data to a json filepageListFilter
: An optional function. It it is present, all output from loadPageList function is passed through this function. Even if filtered output is empty we need not to worry about that, Crawler will manage that by repeated calling of loadPageList function until it gets some data or loadPageList function returns an empty result.onError
: a function function(err)
called upon error. It will not stop the Crawler.onFinish
: a function called upon the completion of whole tasks.concurrency
: no.of parallel requests to processed during scraping.See the examples/blogspot/
for an example crawler that scraps whole posts from a blogspot blog and dumps into a json file.
Depreciated. Browser class which is more simple ( and of course feature less ) instead. We should extend Browser to meet our custom use
Documentation is not yet complete see source code for undocumented features..
FAQs
A general purpose scrapper written using cheerio, request, async etc. It can scrap data from html pages based on some schema defnition. A schema denition includes a css selector and a function that accepts this element and returns the data. It is simple
The npm package html-scrapper receives a total of 2 weekly downloads. As such, html-scrapper popularity was classified as not popular.
We found that html-scrapper demonstrated a not healthy version release cadence and project activity because the last version was released a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
Cloudflare has launched a setup wizard allowing users to easily create and manage a security.txt file for vulnerability disclosure on their websites.
Security News
The Socket Research team breaks down a malicious npm package targeting the legitimate DOMPurify library. It uses obfuscated code to hide that it is exfiltrating browser and crypto wallet data.
Security News
ENISA’s 2024 report highlights the EU’s top cybersecurity threats, including rising DDoS attacks, ransomware, supply chain vulnerabilities, and weaponized AI.