Huge News!Announcing our $40M Series B led by Abstract Ventures.Learn More
Socket
Sign inDemoInstall
Socket

dagmar

Package Overview
Dependencies
Maintainers
1
Versions
3
Alerts
File Explorer

Advanced tools

Socket logo

Install Socket

Detect and block malicious and high-risk dependencies

Install

dagmar

deadly simple crawling/scraping package for Node.

  • 0.1.1
  • latest
  • npm
  • Socket score

Version published
Weekly downloads
0
Maintainers
1
Weekly downloads
 
Created
Source

Build Status Dependency Status devDependency Status

Dagmar

Dagmar is a deadly simple crawling/scraping package for Node.

It features:

  • A clean, simple API
  • Possible use of server-side DOM & automatic jQuery insertion with Cheerio
  • node 0.10+ support

How to install

$ npm install dagmar

Crash course


var crawler = new Crawler();

crawler.forEach(function(error, response, body) {
  if (error || response.statusCode !== 200) {
    console.log(error);
  } else {
    console.log(body);
  }
});

crawler.end(function() {
  console.log('Done.');
});

crawler.queue("http://www.google.com");
crawler.queue("http://www.yahoo.com");
crawler.queue("http://www.apple.com");
crawler.queue("http://www.twitter.com");
crawler.queue("http://www.facebook.com");

crawler.start();

Using Cheerio


var crawler = new Crawler();

crawler.forEach(function(error, response, body) {
  var $, list;
  if (!error && response.statusCode === 200) {
    $ = cheerio.load(body);
    list = $('ul', '<ul id="fruits">...</ul>');
    console.log(list);
  } else {
    console.log(error);
  }
});

crawler.end(function() {
  console.log('Done.');
});

crawler.queue("http://www.fruits.org");

crawler.start();

Full crawler retrieving href and adding to queue

var crawler = new Crawler();

crawler.forEach(function(error, response, body) {
  if (!error && response.statusCode === 200) {
    var $ = cheerio.load(body);
    return $('a').each(function(index, a) {
      var url = $(a).attr('href');
      crawler.queue(url);
    });
  } else {
    console.log(error);
  }
});

crawler.end(function() {
  console.log('Done.');
});

crawler.queue("http://www.fruits.org");

crawler.start();

FAQs

Package last updated on 21 Mar 2016

Did you know?

Socket

Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.

Install

Related posts

SocketSocket SOC 2 Logo

Product

  • Package Alerts
  • Integrations
  • Docs
  • Pricing
  • FAQ
  • Roadmap
  • Changelog

Packages

npm

Stay in touch

Get open source security insights delivered straight into your inbox.


  • Terms
  • Privacy
  • Security

Made with ⚡️ by Socket Inc