harvestd

Components

• harvest daemon (harvestd)
• harvest client
• harvest front-end/visualizer

Run standalone

Environment variables

# webserver
HARVESTD_SERVER_PORT 					# port to run the harvestd webserver on
HARVESTD_SERVER_COOKIE_DOMAIN 			# domain for your cookies 
HARVESTD_SERVER_COOKIE_SECURE 			# make it a secure cookie
HARVESTD_SERVER_COOKIE_MAX_AGE 			# max age
HARVESTD_SERVER_COOKIE_HTTP_ONLY 		# http only?
HARVESTD_SERVER_COOKIE_PATH 			# valid path
HARVESTD_SERVER_ALLOW_CORS 				# allow CORS - useful if your harvestd process is on a different domain

# elasticsearch store
HARVESTD_ELASTICSEARCH_HOST				# elasticsearch host
HARVESTD_ELASTICSEARCH_PORT				# elasticsearch port
HARVESTD_ELASTICSEARCH_API_VERSION		# elasticsearch API version to use
HARVESTD_ELASTICSEARCH_INDEX			# the index to store everything
HARVESTD_ELASTICSEARCH_TYPE				# the name of the type events will be stored as

Install and run

npm install --global harvestd

harvestd

Use as a library

npm install harvestd

var Harvestd = require('harvestd');

var server = Harvestd.create({
	logger: logWranglerInst, 				// optional paramter to provide your own logwrangler object
	store: customStore 						// optional paramter to provide your own store interface
	config: {
		cookieDomain: 'mydomain.com',		// defaults to localhost
		cookieSecure: false, 				// defaults to false
		cookieMaxAge: 365 * 60 * 24 * 24,	// defaults to 1 year (in seconds)
		cookieHttpOnly: true 				// defaults to true
		allowCORS: true 					// defaults to true
	},
	server: express() 						// optionally provide an express server instance
});

server.start(); 						// If you dont want the default port (9000) you can pass one as the 
										// first argument to "server.start(8080)"

Creating a custom store

By default, Harvestd comes with and Elasticsearch adapter and will use it as its default data store. Depending on your usecase however, you may want to use some other datastore, multiple datastores, or event format your data in an entirely different way. To do that, all you need to do is implement the Harvestd.Store prototype for track and identify and pass an instance of it when creating your Harvestd server instance.

var Q = require('q');
var Harvestd = require('harvestd');

function MyStore(){
	// do whatever initialization you want
}

MyStore.prototype = Object.create(Harvestd.Store.prototype);

/**
	@string token		Account identification token (if not running multi-tenant, 
						this can be any string value with a length >= 1). Otherwise
						You should use this value to identify each user/account.
		
	@string event 		The event name
	@object data		Object literal representing the data that needs to be inserted
*/
MyStore.prototype.track = function(token, event, data){
	// Impelment me

	// must return a thennable promise
	return Q.resolve();
};


/**
	@string token		Account identification token (if not running multi-tenant, 
						this can be any string value with a length >= 1). Otherwise
						You should use this value to identify each user/account.

	@string uuid 		The uuid to use to find events to set the userId on
	@string userId		The userId to set
*/
MyStore.prototype.identify = function(token, uuid, userId){
	// Impelment me

	// must return a thennable promise
	return Q.resolve();
};

var server = Harvestd.create({
	store: new MyStore() 			// optional paramter to provide your own store interface
});

server.start();

Available helper functions in the Store prototype:

Store.prototype.formatAttribute(str)

Takes an attribute and formats it. If you pass and object or an array, it will traverse it, formatting all nested fields.

The main purpose is to ensure that values are the correct types. For example:

	var data = {
		someBool: "true"
	};

	data = Store.prototype.formatAttribute(data);

	/*
		The string "true" is converted into an actual boolean value. 
		{
			someBool: true
		}
	*/

ESStore Storage Layer

Harvestd comes with a really simple layer that will store everything in Elasticsearch.

Creation

var Harvestd = require('harvestd');

var store = Harvestd.ESStore.create({
	host: '127.0.0.1', 		// defaults to localhost
	port: 9200, 			// defaults to 9200
	apiVersion: '1.1', 		// defaults to the latest ES API version
	index: 'analytics', 	// the index to store everything. Defaults to "analytics"
	type: 'event' 			// the name of the type things will be stored as. Defaults to "event"
});

var server = Harvestd.create({
	store: store
});

server.start();

API Routes

Track

Track an event with any arbitrary amount of data. See services/harvestd/modules/api/elasticsearchStore/mappings for an example of available fields (if using the ESStore handler).

POST /track
Content-Type: application/json

{
	"token": "your account token",
	"event": "an event name",
	"data": {
		"$uuid": "client uuid"
	}
}

Identify

Identify is used to associate tracked events with a particular UUID with some kind of userId from your own backend. This can be any string value you want - an integer user id, a user's email, etc, as long as its unique within your system. By default when you track an event, the $uuid will be used as the $userId if the $userId is not provided.

Calling identify() will also store the provided userId for the session so subsequent track() calls are identified.

identify() should be called whenever a user can be identified; this could be after signup, login, or at any other point that a user's identity is determined in your app.

POST /identify
Content-Type: application/json

{
	"token": "your account token",
	"uuid": "client uuid"
	"userId": "ID of identified user"
}

Customer Profiles

Customer profiles allows you to store information about your customers. Customers need to have some kind of unique identifier - this can be an internal id from your own database, and email, etc.

Due to the possibility of users interacting with more than one platform, it is possible for multiple session UUIDs (used in the track and identify functions). This will help you identify actions taken by each user regardless of platform/location.

POST /profile
Content-Type: application/json

{
	"token": "your account token",
	"$id": "customer unique id",
	"uuid": "their current UUID"
}

Using the web client

Include the client on your page:

<script type="text/javascript" src="//<domain where your harvestd instance is>/js/client.js"></script>
<script type="text/javascript">
	
</script>