etp

etp-js

This is a javascript binary serialization implementation for the Energistics Transfer protocol (ETP). ETP is a proposed specification for streaming real-time data from oil field drilling and production facilities. It uses websockets for transport and Apache Avro for serialization. Note that there is no need for a libary to serialize the JSON versions of objects, since JSON.parse and JSON.stringify can be used directly with the websocket messages. However, the schema validation methods described below can also be used with JSON objects.

This node package contains:

1. A set of routines for serializing and deserialing binary Avro messages in javascript (included from the etp-avro package).
2. An array of the all of the ETP protocol message schemas in javascript form (i.e. a parsed version of the JSON message schemas) which can be passed to the Avro serializer to parse an incoming message.
3. Concrete javascript classes for the schemas in a release.

Prerequisites

• Install Node from nodejs.org - v0.10 min required.

Installation

To install from npm

Note that a previous version of this library exists called node-etp and has been deprecated.

npm install etp

To install from source

Clone the node folder from bitbucket.

c:\energistics\src git clone https://bitbucket.org/energistics/node-etp.git

Install into your project directory c:\ralfdemo>npm install c:\energistics\src\node-etp

Using the library

From the node command prompt:

load the library

> var etp = require("etp");	

the ETP library wraps the etp-avro package. From this we can get readers and writers, etc.

> var avro = etp.avro;

Create a new schema cache object. This is used by the library to read, write, and validate schemas. Note the use of etp.SchemaCache() vs. avro.SchemaCache(schemas). The etp version of the SchemaCache class automatically loads the correct schemas, and has utility functions to deal with ETP messages.

>var sc = new etp.SchemaCache();

The following finds the schema associated with protocol 1, message 1. You can use it on incoming messages after messages, after decoding the header.

> sc.find(1, 1);
{ type: 'record',
  namespace: 'Energistics.Protocol.ChannelStreaming',
  name: 'ChannelDescribe',
  messageType: '1',
  protocol: '1',
  fields: [ { name: 'uris', type: [Object] } ],
  fullName: 'Energistics.Protocol.ChannelStreaming.ChannelDescribe',
  depends: [] }
>

Reading a datum

The decode function on the reader allows you to pass any array of values to decode a single datum (which can also be a structured type, so long as it is registered in the schema cache).

> var reader = new avro.BinaryReader(sc);
> var msg = reader.decode("string", [22, 0x48, 0x65, 0x6c, 0x6c, 0x6f, 0x20, 0x57, 0x6f, 0x72, 0x6c, 0x64]);
> console.log(msg);
Hello World

However, assuming you have received a binary message from a websocket object. You can attach the binary message to the reader by using the builtin js type Uint8Array. You can then reader the message header first, and then lookup the appropriate schema for the message boday.

reader = new avro.BinaryReader(sc, new Uint8Array(message.binaryData)),
header = reader.readDatum("Energistics.Datatypes.MessageHeader"),
body = reader.readDatum(this.schemaCache.find(header.protocol, header.messageType));

Writing a datum

The writing functions work more or less the same way. The getBuffer() method gets the value which can be passed directly to the websocket send() method.

writer = new avro.BinaryWriter(sc);
writer.writeDatum("Energistics.Datatypes.MessageHeader", header);
writer.writeDatum(schemaName, message);
dataToSend = encoder.getBuffer();

Known limitations

2^53 is the largest effective integer value that we can put in a javascript Number type (which is a 64 bit IEEE double precision number) without loss of precision. Negative is 1 power of 2 smaller. After this point, avro longs will lose precision.

    [128, 128, 128, 128, 128, 128, 128, 32] ==> 9007199254740992
    [255, 255, 255, 255, 255, 255, 255, 15] ==> -4503599627370496

Validation

The schema cache is itself an associative array of schemas

> sc['Energistics.Datatypes.Version']
{ type: 'record',
  namespace: 'Energistics.Datatypes',
  name: 'Version',
  fields:
   [ { name: 'major', type: 'int' },
     { name: 'minor', type: 'int' },
     { name: 'revision', type: 'int' },
     { name: 'patch', type: 'int' } ],
  fullName: 'Energistics.Datatypes.Version',
  depends: [] }
>

The schema cache object can be used to validate JS objects. Validate does not return a value, simply throws if the input object does not match the schema. In testing and debugging phases, especially when you implement a new version of the protocol, validation is recommended for all messages.

> sc.validate('Energistics.Datatypes.Version', {major: 11, minor: 2, revision:3, patch:4});
undefined
> sc.validate('Energistics.Datatypes.Version', {major: 11, minor: 2, revision:3, patch:"string"});
TypeAssertionError: expected integer
    at D:\test\etp\node_modules\etp\node_modules\etp-avro\lib\assert-type.js:212:13
    at SchemaCache.validate (D:\test\etp\node_modules\etp\node_modules\etp-avro\lib\etp-avro.js:586:42)
    at SchemaCache.validate (D:\test\etp\node_modules\etp\node_modules\etp-avro\lib\etp-avro.js:599:26)
    at SchemaCache.validate (D:\test\etp\node_modules\etp\node_modules\etp-avro\lib\etp-avro.js:648:22)
    at repl:1:4
    at REPLServer.self.eval (repl.js:110:21)
    at repl.js:249:20
    at REPLServer.self.eval (repl.js:122:7)
    at Interface.<anonymous> (repl.js:239:12)
    at Interface.emit (events.js:95:17)

As noted above, you can use any old javascript object with the serializers and validation routines, but the library also provides concrete classs, within namespaces. The concrete classes initialize a hash with reasonable defaults for most fields, including any Avro default that is specified. They also keep a copy of the schema itself as the _schema member of the hash.

> var version = new etp.Messages.Energistics.Datatypes.Version();
undefined
> version
{ major: 0,
  minor: 0,
  revision: 0,
  patch: 0,
  _schema:
   { type: 'record',
     namespace: 'Energistics.Datatypes',
     name: 'Version',
     fields:
      [ [Object],
        [Object],
        [Object],
        [Object] ],
     fullName: 'Energistics.Datatypes.Version',
     depends: [] } }
>

Using in a browser

To use in a web browser application, simply copy either the webified or webified/minified version of the .js file from the node_modules/etp/lib/browser to an appropriate website folder and include this file a script tag.

<script type="text/javascript" src="js/etp.min.js"></script>

You can get it from the jsdelivr CDN as use in the code below.

Loading the browser modules creates a property on the window object with a full export of the library classes.

#!html

<html>
<head>
    <meta charset="utf-8">
    <title>ETP Test</title>
</head>
<body style="font-family: verdana; font-size: 9pt; color: black">
<div id="etp"></div>
<script src="http://cdn.jsdelivr.net/etp/latest/etp.min.js"></script>
<script>
	function setup() {
		var etp = window.etp;
		// List all schema types
		for (var i=0; i<etp.Schemas.types.length; i++) {
			document.getElementById("console").innerHTML+= ( "<br/>"   + etp.Schemas.types[i].fullName );
		}

		// de-serialize a binary avro array
		var avro = etp.avro;
		var schemaCache = new avro.SchemaCache( [] );
		var reader = new avro.BinaryReader(schemaCache);
		var msg = reader.decode("string", [22, 0x48, 0x65, 0x6c, 0x6c, 0x6f, 0x20, 0x57, 0x6f, 0x72, 0x6c, 0x64]);
		document.getElementById("console").innerHTML+= ( "<br/><br/>Binary decode: "   + msg );
		

	}
	
    window.onload = setup;
</script>
Hello ETP
<div id="console" />
</body>
</html>