scrapyjs-jsdom-server
scrapyjs-jsdom-server
is a Scrapyjs-compatible jsdom-based rendering service.
This Node-based server is a drop-in replacement for the default server that's used with Scrapyjs.
Purpose
This project was started because the default Splash server that ScrapingHub makes almost always crashes after a few hours of heavy scraping (memory bug perhaps?).
Try using the default Splash server and if you find that you can't use it for heavy scraping, come back and use this server instead. Happy scraping!
Usage
Standalone
git clone https://github.com/paulborza/scrapyjs-jsdom-server.git
cd scrapyjs-jsdom-server
npm install
cd src
supervisor app.js
As a Node package
npm install --save scrapyjs-jsdom-server