Scraper API

Llama client to pull metrics from Llama collectors LLAMA Collector sends UDP probes to a set of target reflectors and provides statistics about their latency and reachability via an API. Mock Llama client used in `client_test.go` Functionality for sending and receiving UDP probes on a socket. portgroup defines PortGroup, which is used to multiplex UDPAddr structs to multiple ports via parallel channels. LLAMA Scraper pulls stats from Collectors and then writes them to the indicated database. Tags is a helper description for a structure that stores a map of attributes and values for a given key. Example: Tags["1.2.3.4"]["dst_hostname"] = "localhost" Tags["1.2.3.4"]["dst_cluster"] = "mycluster"

Dtoo exposes an HTML scraper API inspired by the artoo.js Scrape API https://medialab.github.io/artoo/scrape/. The dtoo scrape API closely follows artoo's scrape API, but is slightly modified to suit Go. The biggest changes are that "scrapeTable" is not implemented and "scrapeOne" is implemented via the ScrapeFromXxxWithLimit functions. The artoo example written for dtoo would look like this. The above example will return a slice of dtoo.Model objects each with the following keys: {id, content}. The dtoo data model passed to the ScrapeXxx and ScrapeXxxWithLimit functions can be a string, func (s *goquery.Selection) (interface{}, error), dtoo.Model or dtoo.RetrieverModel. Retrieves a slice of post id attributes using a string data model. Retrieves a slice of post comments using a function data model. When using a function you are exposed to the low level goquery API for scraping out content from the DOM. Retrieves a slice of dtoo.Model objects each with the following keys {Id, Title, PublishDate}. Using dtoo.Model you can nest dtoo.RetrieverModel objects and functions as keys in your model to retrieve infinitely complex models. This example retrieves a slice of dtoo.Model objects with the following properties {Id, Title, PublishedDate, Comments: [{Id, Author, Content}]}. The above example could be alternatively achieved by using the recursive Scrape setting of RetrieverModel.