deep-scrape

Scrape and crawl pages with io.js and get a whole lot of meta data. Shows; headers, Ajax requests/responses, rendered html, Javascript AST's, dependencies, console events, and a whole lot more. Crawl sites, or scrape a single page. Add cookies or proxy re

0.0.5

latest

Source

npm

Version published: 9 years ago

Maintainers: 1

Created: 10 years ago

Source

Deep Scrape

Installation

This was tested on node 0.12.x. It can be run as a module export, or a command line script.

npm install deep-scrape
// or clone the repository and run it as a script.

Use Case

You are scraping websites with lots of javascript (Angular, Ember, Browserfy).
You don't mind trading a bit of performance for more detailed scraping data.
You would like to find potential DOM sinks and sources on your pages (Possibly for vulnerability scanning).
You need the most detailed metadata, metrics, and analyitics on your scraped pages.
You would like to fingerprint possible technologies a certain site or page uses.

Keywords

FAQs

What is deep-scrape?

Is deep-scrape well maintained?

Package last updated on 06 May 2016

Did you know?

Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.

Install