JEDI CRAWLER
Da fuq?
JEDI CRAWLER is a Node/PhantomJS crawler made to scrape pretty much anything from Node, with a really simple syntax. Work in progress ladies
How does it work
Register padawans to the jedi crawler, that have a pattern to match URL, and jQuery-style selectors. You can also post-process the data.
module.exports = function(jedi) {
jedi.registerPadawan({
pattern: /en.wikipedia.org\/wiki\//,
selectors:{
title:{
sel: "#firstHeading span",
type: "text"
},
firstParagraph:{
sel: "#toc ~ p:first",
type: "text"
}
},
postProcessing: function(data) {
data.title = data.title.toUpperCase();
return data;
}
});
};
For now only two types of selectors are supported : "text" and "src"
I find having one file per padawan (crawler) pretty cool for code clarity and also padawans need to learn by themselve and be alone
You can then give your padawans to the Jedi by doing
var jedi = require('./modules/jedi');
require('./padawans/wikipedia')(jedi);
And then you can do
jedi.crawl('http://en.wikipedia.org/whatever', function(err, result){
console.log(err);
console.log(result);
});
Special features
Crawlers only start to scrape the page as soon as $(document).ready is fired. Our own version of jQuery is injected into the page, but then we also give back the $ to its owner in case they're executing 3rd party libraries to modify the DOM or w/e
If your selectors matches severals DOM elements, then an array of every value is returned
Right now, PhantomJS is instantiated with "--load-images=no" option so the page loads faster
Test it now
Pull that bad boy
Make sure you have PhantomJS installed
Run node main.js