loopback-connector-screenscrape

Screen Scraping Connector for Loopback

NOTE: This is experimental!

This is a quick sample connector for Loopback using https://github.com/lapwinglabs/x-ray to screen scrape a website and return the data through a LB model.

Example App

An example app is provided here: https://github.ibm.com/apiconnect-solution-architecture/example-screenscrape-app

Usage

This connector requires two properties:

website: The URL of the page you want to scrape.

domTarget: The root DOM node from which to start the screen scrape.

Example:

{
	"dribbble": {
		"connector": "screenscrape",
		"name": "dribbble",
		"remotingEnabled": true,
		"website": "https://dribbble.com/search?q=mobile",
		"domScope": "li.group"
	}
}

For the model field definitions, you must tell the connector what DOM target this field will be bound to. e.g.

"properties": {
	"title": {
		"type": "string",
		"domTarget": ".dribbble-img strong"
	},
	"image": {
		"type": "string",
		"domTarget": ".dribbble-img [data-src]@data-src"
	},
	"comment": {
		"type": "string",
		"domTarget": ".dribbble-img .comment"
	},
	"date": {
		"type": "type",
		"domTarget": ".dribbble-img .timestamp"
	},
	"commentCount": {
		"type": "string",
		"domTarget": ".dribbble-shot .tools .cmnt span"
	},
	"viewsCount": {
		"type": "string",
		"domTarget": ".dribbble-shot .tools .views span"
	},
	"likes": {
		"type": "string",
		"domTarget": ".dribbble-shot .tools .fav a"
	}
},

Tests

If you want to try it out quickly, as a stand-alone test:

• Download this repo and npm install
• Run node example/model.js
• You'll see results from https://dribbble.com/search?q=mobile in JSON format. Checkout example/model.js for a quick example.

To run the actual test(s): npm test