@ably/delta-codec

Vcdiff Codec Library for JavaScript

A Vcdiff codec library supporting the Vcdiff delta format, as defined by RFC 3284. This library supports developers who need to consume delta streams from Ably without using the official JavaScript client library (e.g. for MQTT and SSE applications).

Throughout this documentation, and within the code itself, we refer to a Vcdiff payload as a 'delta'. Elsewhere such delta payloads may be referred to as patches or diffs, but for consistency within this repository we stick to the terms 'delta' and 'deltas'.

Installation from npm for Node.js

npm install @ably/delta-codec

and require as:

var deltaCodec = require('@ably/delta-codec');

Basic Stream Decoder

The VcdiffDecoder constructor provides the most basic entry point to the public API. It provides a stateful way of applying a stream of Vcdiff deltas, producing a new value after each delta has been applied to the previous value.

First provide the base value, upon which the first delta will be applied using the 'setBase' method:

let decoder = new deltaCodec.VcdiffDecoder();
decoder.setBase(value);

Once the decoder has been initialized like this, then each subsequent delta is applied using the applyDelta method:

let result = decoder.applyDelta(delta);
// TODO call method on result to get the value format you require

The decoder retains the current value, with the result of this method encapsulating that new value now that delta has been applied to the previous value. The result of this method offers methods to allow you to access the new current value in the format you require:

• asUint8Array(): a Uint8Array, being the new current value, as received (i.e. 'raw' bytes)
• asUtf8String(): a string, decoded from data using UTF-8
• asObject(): a JavaScript object, decoded from data using JSON.parse

setBase(value) Instance Method

Accepts a single value argument, the 'base', which may be ArrayBuffer, Uint8Array, Buffer or string. If a string is supplied then it will be UTF-8 encoded by the library before being stored as the current value. This is freeform data, as specified by the application.

The decoder also exposes an alternative method, setBase64Base(value), where the single value argument must be string and is Base64 decoded by the library before being stored as the current value.

applyDelta(delta) Instance Method

Accepts a single delta argument which may be ArrayBuffer, Uint8Array or Buffer. This is a Vcdiff format delta.

The decoder also exposes an alternative method, applyBase64Delta(delta), where the single delta argument must be string and is Base64 decoded by the library before being applied as a Vcdiff format delta to the current value.

isDelta(data) Static Method

Accepts a single data argument which may be ArrayBuffer, Uint8Array or Buffer. Returns true only if data has a Vcdiff delta header.

This method can be used on receipt of a binary payload to detect whether it should be interpreted as an absolute value or as a delta to be applied to the previous value. Such 'sniffing' should be avoided where there is metadata available alongside received payloads to indicate whether they are deltas or not, as is the case when receiving a stream of enveloped data from Ably over SSE (see example).

Checked Stream Decoder

The CheckedVcdiffDecoder is a variant of VcdiffDecoder that can be used when the values and the deltas applied to them have unique identifiers. The 'set' and 'apply' methods on the checked decoder have the same names but require additional arguments for these identifiers:

• applyDelta(delta, deltaId, baseId)
• applyBase64Delta(delta, deltaId, baseId)
• setBase(value, baseId)
• setBase64Base(value, baseId)

The baseId argument supplied to the 'set' methods and the deltaId arguments supplied to the 'apply' methods are stored alongside the current value and then compared to the baseId argument supplied in subsequent 'apply' calls. An Error is thrown if there is a mismatch.

Example Use Cases

Node.js: Text stream from Ably via SSE (enveloped)

By default the event data received from Ably is enveloped in JSON format. We decode this data and inspect the Ably formatted message contents in order to establish whether the data in this message is an absolute value or a delta to be applied to the previously received value.

const deltaCodec = require('@ably/delta-codec');
const EventSource = require('eventsource');

const prefix = '[?delta=vcdiff]';
const url = `https://realtime.ably.io/event-stream?channels=${prefix}${CHANNEL_NAME}&v=1.2&key=${APP_KEY}`;
const eventSource = new EventSource(url);
const decoder = new deltaCodec.CheckedVcdiffDecoder();
eventSource.onmessage = function onEventSourceMessage(event) {
  const message = JSON.parse(event.data);
  let value;
  const deltaExtras = (message.extras && message.extras.delta) ? message.extras.delta : null;
  if (deltaExtras) {
    if (deltaExtras.format !== 'vcdiff') {
      throw new Error(`Delta format ${deltaExtras.format} not understood.`);
    }
    value = decoder.applyBase64Delta(message.data, message.id, deltaExtras.from).asUtf8String();
  } else {
    value = message.data;
    decoder.setBase(value, message.id);
  }
  console.log(`received: ${value}`);
};
eventSource.onerror = function onEventSourceError(event) {
  console.log(`error: ${event.data}`);
};

Node.js: Text stream from Ably via SSE (not enveloped)

For this example we have subscribed to Ably as our event source and specified that we do not want the inbound event data to be JSON enveloped. Without envelopes the events will be smaller, taking up less transmission bandwidth, however this then means we need 'sniff' each inbound event's data to identify whether it is an absolute value or a delta to be applied to the previously received value.

Absolute values are sent to us as strings, ready to use. Deltas are sent to us as Base64 encoded binary.

const deltaCodec = require('@ably/delta-codec');
const EventSource = require('eventsource');

const prefix = '[?delta=vcdiff]';
const url = `https://realtime.ably.io/event-stream?channels=${prefix}${CHANNEL_NAME}&v=1.2&key=${APP_KEY}&enveloped=false`;
const eventSource = new EventSource(url);
const decoder = new deltaCodec.VcdiffDecoder();
eventSource.onmessage = function onEventSourceMessage(event) {
  const stringData = event.data;
  let value;
  if (deltaCodec.VcdiffDecoder.isBase64Delta(stringData)) {
    value = decoder.applyBase64Delta(stringData).asUtf8String();
  } else {
    value = stringData;
    decoder.setBase(value);
  }
  console.log(`received: ${value}`);
};
eventSource.onerror = function onEventSourceError(event) {
  console.log(`error: ${event.data}`);
};

Node.js: Binary stream from Ably via MQTT

The raw binary data received over MQTT has no encoding or other form of envelope encapsulating it. We need to 'sniff' each inbound payload to identify whether it is an absolute value or a delta to be applied to the previously received value. In this example we are transporting UTF-8 encoded strings.

const deltaCodec = require('@ably/delta-codec');
const mqtt = require('mqtt');

const brokerUrl = `mqtts://mqtt.ably.io`;
const options = {
  username: APP_KEY_NAME,
  password: APP_KEY_SECRET,
};
const prefix = '[?delta=vcdiff]';
const client = mqtt.connect(brokerUrl, options);
client.on('connect', () => {
  client.subscribe(`${prefix}${CHANNEL_NAME}`);
});
const decoder = new deltaCodec.VcdiffDecoder();
client.on('message', (topic, message) => {
  let value;
  if (deltaCodec.VcdiffDecoder.isDelta(message)) {
    value = decoder.applyDelta(message).asUtf8String();
  } else {
    decoder.setBase(message);
    value = message.toString();
  }
  console.log(`received: ${value}`);
});

Contributing

Building

You can trigger a build using Webpack with:

npm run grunt -- build

which creates delta-codec.js and delta-codec.min.js in the dist folder.

Testing

To run tests in all runtimes (Node and browsers):

npm test

To run tests on a single runtime:

• Node (very quick): npm run grunt -- test:node
• Local browser (Firefox): npm run grunt -- test:browser:local
• Remote browsers (Safari, Firefox, Chrome, IE, Edge, Chrome Mobile and Mobile Safari): npm run grunt -- test:browser:remote

Known Issue: When testing in a local browser either using npm run grunt -- test:browser:local or indirectly by using npm test on macOS, you may see a "segmentation fault". Launch the Console app to find the associated crash reports, of which there will be too. More information in issue #7.

Remote browser testing supported by

for which you will need to configure environment variables for BROWSERSTACK_USERNAME and BROWSERSTACK_ACCESSKEY.

Release Procedure

On the master branch:

• Increment the version, regenerate from source (a.k.a. build / bundle) and make a tagged commit which includes the built output from the /dist folder by running npm run grunt -- release:patch (or "major", "minor" or "prepatch" as appropriate - see grunt-bump Usage Examples)
• Release the tagged commit to Github using git push origin master --follow-tags
• Release to NPM using npm publish . --access public (this package is configured to require that 2FA is used by publishers)
• Release to Ably's CDN using npm run grunt -- publish-cdn (operable by Ably staff only)
• Visit tags and draft new release for the newly created tag