thriftrw

What is thriftrw?

The thriftrw npm package is a Thrift IDL (Interface Definition Language) compiler and runtime for JavaScript. It allows you to define data types and service interfaces in Thrift IDL and then generate JavaScript code to serialize, deserialize, and handle these types and services.

What are thriftrw's main functionalities?

Serialization and Deserialization

This feature allows you to serialize and deserialize Thrift objects. The code sample demonstrates how to define a Thrift struct, create an instance of it, serialize it to a buffer, and then deserialize it back to an object.

const Thrift = require('thriftrw').Thrift;
const thrift = new Thrift({source: 'struct User { 1: string name, 2: i32 age }'});
const User = thrift.User;

const user = new User({name: 'Alice', age: 30});
const buffer = User.toBuffer(user);
const deserializedUser = User.fromBuffer(buffer);
console.log(deserializedUser);

Service Interface

This feature allows you to define and implement Thrift service interfaces. The code sample demonstrates how to define a Thrift service, implement it, and call a method on the service.

const Thrift = require('thriftrw').Thrift;
const thrift = new Thrift({source: 'service UserService { User getUser(1: i32 id) } struct User { 1: string name, 2: i32 age }'});
const UserService = thrift.UserService;

const userService = new UserService({
  getUser: function (id, callback) {
    callback(null, new thrift.User({name: 'Alice', age: 30}));
  }
});

userService.getUser(1, (err, user) => {
  if (err) throw err;
  console.log(user);
});

Other packages similar to thriftrw

thrift

The 'thrift' npm package is the official Apache Thrift library for JavaScript. It provides similar functionalities to thriftrw, including serialization, deserialization, and service interface handling. However, thriftrw is designed to be more lightweight and faster, focusing on runtime performance and ease of use.

node-thrift

The 'node-thrift' package is another implementation of the Thrift protocol for Node.js. It offers similar features such as serialization, deserialization, and service interface support. Compared to thriftrw, node-thrift is more mature and widely used in production environments, but it may not be as performant or easy to use as thriftrw.

README

thriftrw

Encodes and decodes Thrift binary protocol and JavaScript object models declared in a Thrift IDL. This is an alternative approach to using code generated from Thrift IDL source files with the thrift compiler.

ThriftRW supports encoding and decoding the protocol with or without Thrift IDL sources. Without sources, it is still possible to read and write the Thrift binary protocol in all of its structure using numbered fields for structs and not discerning the nuanced differences between binary and string, list and set, or number and enum.

With a Thrift IDL, ThriftRW is able to perform certain optimizations. ThriftRW constructs model instances directly from the contents of a buffer instead of creating an intermediate anonymous structural model. As a consequence, it is able to quickly skip any unrecognized fields. ThriftRW can also use the same struct constructor for every instance of a struct, which should yield run-time performance benefits for V8 due to hidden classes (not yet verified).

The scope of this project may eventually cover alternate Thrift binary encodings including the compact binary protocol.

This project makes extensive use of bufrw for reading and writing binary protocols, a component shared by tchannel-node, with which this library works in concert.

Example

ThriftRW provides a Thrift constructor that models all of the services, functions, and types expressed in a ThriftIDL source file.

var fs = require('fs');
var path = require('path');
var Thrift = require('thriftrw').Thrift;
var source = fs.readFileSync(fs.path(__dirname, 'meta.thrift'), 'ascii');
var thrift = new Thrift({source: source, strict: true});

Consider meta.thrift

struct HealthStatus {
    1: required bool ok
    2: optional string message
}

service Meta {
    HealthStatus health()
    string thriftIDL()
}

The most common usage of a Thrift instance is to get the argument and result types of a function and use those types to read to or write from a buffer. The getType and getTypeResult functions retrieve such models based on their name. For example, the arguments struct for the health endpoint would be Meta::health_args and its result struct would be Meta::health_result.

var MetaHealthArgs = thrift.getType('Meta::health_args');
var MetaHealthResult = thrift.getType('Meta::health_result');

The getType method will return the struct or throw an exception if one does not exist. This is the appropriate method if getType should throw an error due to a programmer's mistake, where the Thrift source in question is checked into the same project and the method name is a literal.

However, exceptions being an undesirable outcome for bad data from a user, getTypeResult returns a result object with either err or value properties set.

var res = thrift.getTypeResult(methodName + '_args');
if (res.err) {
    return callback(res.err);
}
var ArgsStruct = res.value;

The struct can be written or read to or from a buffer using the struct's rw instance. A RW (reader/writer) implements byteLength(value), writeInto(value, buffer, offset), readFrom(buffer, offset), each of which return a result object as specified by bufrw. The value may be JSON, a POJO (plain old JavaScript object), or any instances of the Thrift model, like new thrift.Health({ok: true}).

A struct can also be encoded or decoded using the ThriftStruct's own toBuffer(value), fromBuffer(buffer), toBufferResult(value), and fromBufferResult(buffer) methods. Those with Result in their name return Result instances instead of throwing or returning the buffer or value.

TChannelASThrift employs this interface on your behalf, leaving the application author to take arguments and call back with a result when using the Thrift argument scheme and a given Thrift IDL.

Without Thrift IDL

ThriftRW provides T prefixed types for encoding and decoding the wire protocol without Thrift IDL. This is useful for applications like [TCap] that do not necessarily have a Thrift IDL file on hand to make sense of structs by field numbers alone.

The following example illustrates reading and writing a struct. ThriftRW exports TStruct, TList, and TMap. TStructs serve for arguments, results, and exceptions. TLists also serve for sets.

var thriftrw = require("thriftrw");
var bufrw = require('bufrw');

var struct = new thriftrw.TStruct();
struct.fields.push(
    new thriftrw.TField(thriftrw.TYPE.STRING, 1, new Buffer('hello')
);

var buf = bufrw.toBuffer(thriftrw.TStructRW, struct);
console.log('created a binary buffer of thrift encoded struct', buf);

var struct2 = bufrw.fromBuffer(thriftrw.TStructRW, buf);
console.log('created a TStruct from a binary buffer', struct2);

Thrift Model

Thrift has internal models for modules, services, types, and values. These models are indexed by name on each Thrift module in the models object. Each of these models has a "surface", which is a constructor for types, values for constants, an index of functions for services, and the model itself for modules. Modules expose their names on their own object only if they start with a non-lower-case character, to avoid name collisions with other properties and methods, but are always also available through the index for their type class, like thrift.consts.PI.

The Meta service is simply thrift.Meta. The object is an object mapping functions by name, so thrift.Meta.health is the interface for accessing the args and result structs for the Meta::health function.

var args = new thrift.Meta.health.Arguments({ok: true});
var result = new thrift.Meta.health.Result({success: null});

The Thrift instance also has a services property containing the actual ThriftService instance indexed by service name. ThriftService instances are not expected to be useful outside the process of compiling and linking the Thrift IDL, but you can use the services object to check for the existence of a service by its name.

Structs

ThriftStruct models can be constructed with an optional object of specified properties. ThriftRW exposes the constructor for a struct declaration by name on the thrift instance, as in thrift.Health for the meta.thrift example.

var HealthStruct = thrift.Health;
var result = new ResultStruct({
    success: new HealthStruct({ok: true })
})

Unspecified properties will default to null or an instance of the default value specified in the Thrift IDL. Nested structs can be expressed with JSON or POJO equivalents. The constructors perform no validation. Invalid objects are revealed only in the attempt to write one to a buffer.

var result = new ResultStruct({success: {ok: true}});

Each constructor has a rw property that reveals the reader/writer instance. RW objects imlement byteLength(value), readFrom(buffer, offset), and writeInto(value, buffer, offset). The value may be any object of the requisite shape, though using the given constructors increases the probability V8 optimization.

Each constructor also hosts toBuffer(value), fromBuffer(buffer), toBufferResult(value), and fromBufferResult(buffer).

Structs are indexed by name on the thrift.structs object and aliased on the thrift object if their name does not start with a lower-case letter.

Exceptions

ThriftException extends ThriftStruct. Exceptions are modeled as structs on the wire, and are modeled as JavaScript exceptions instead of regular objects. As such they have a stack trace. ThriftRW exposes the exception constructor by name on the thrift instance.

exception Pebcak {
    1: required string message
    2: required string keyboardName
    3: required string chairName
}

var error = new thrift.Pebcak({
    message: 'Problem exists between chair and keyboard',
    chairName: 'Hengroen',
    keyboardName: 'Excalibur'
});

Exceptions are indexed by name on the thrift.exceptions object and aliased on the thrift object if their name does not start with a lower-case letter.

Unions

ThriftUnion also extends ThriftStruct. Unions are alike to structs except that fields must not be marked as optional or required, cannot have a default value, and exactly one must be defined on an instance. As with other types, validation of a union only occurs when it is read or written.

union CoinToss {
    1: Obverse heads
    2: Reverse tails
}

struct Obverse {
    1: required string portrait;
    2: required i32 year;
}

struct Reverse {
    1: required string structure;
    2: optional string motto;
}

var coinToss = new thrift.CoinToss({
    head: thrift.Obverse({
        portrait: 'TK',
        year: 2010
    })
})

Unions are indexed by name on the thrift.unions object and aliased on the thrift object if their name does not start with a lower-case letter.

Enums

Thrift enumerations are surfaced as an object mapping names to strings on the thrift instance.

enum CoinToss {
    tails = 0
    heads = 1
}

var result = Math.random() < 0.5 ?
    thrift.CoinToss.heads :
    thrift.CoinToss.tails;
// result will be either "tails" or "heads"

ThriftRW hides the detail of which numeric value an enumeration will have on the wire and allows an enumeration to be represented in JSON by name. This is a deliberate departure from the norms for Thrift in other languages to take advantage of the dynamic nature of JavaScript, permit duck typing, enforce loose coupling between application code and wire protocol, and make Thrift expressible as JSON on the command line with TCurl

Enums are indexed by name on the thrift.unions object and aliased on the thrift object if their name does not start with a lower-case letter.

Consts

Thrift constants are surfaced as properties of the thrift instance.

const PI = 3
const TAU = 6

var PI = thrift.PI;
var TAU = thrift.TAU;

Consts are indexed by name on the thrift.consts object and aliased on the thrift object if their name does not start with a lower-case letter.

Enhancements and Departures from Thrift proper

ThriftRW operates in a "strict" mode by default that imposes additional constraints on Thrift to ensure cross language interoperability between JavaScript, Python, Go, and Java, the four Thrift bindings favored at Uber.

In strict mode, structs must explicitly declare every field to be either required, optional, or have a default value.

ThriftRW supports forward references and self-referential types (like trees) without special demarcation. ThriftRW also supports forward references in default values and constants.

Types and Annotations

ThriftRW respects certain annotations specific to the treatment of various types in JavaScript.

set

There is an emerging standard for the representation of sets in JavaScript. Until that standard becomes pervasive, sets are traditionally represented either as an array with externally enforced constraints, or as long as the values are scalar like numbers and strings, as an Object with values for keys.

By default, ThriftRW encodes and decodes sets as arrays. ThriftRW does nothing to enforce uniqueness.

Additionally, the js.type annotation can be set to "object" if the value type is a string, i16, or i32. ThriftRW will ensure that the keys are coerced to and from Number for numeric value types.

typedef set<string> (js.type = 'object') StringSet
typedef set<i32> (js.type = 'object') I32Set

map

There is also an emerging standard for the representation of maps in JavaScript. However, until support is more pervasive, ThriftRW supports representing maps as either an object with scalar key types or an entries array for all others.

Maps are represented as objects by default. With the js.type annotation set to "entries", the map will be encoded and decoded with an array of [key, value] duples.

typedef map<string, string> Dictionary
typedef map<Struct, Struct> (js.type = 'entries') StructsByStructs

i64

JavaScript numbers lack sufficient precision to represent all possible 64 bit integers. ThriftRW decodes 64 bit integers into a Buffer by default, and can coerce various types down to i64 for purposes of expressing them as JSON or plain JavaScript objects.

• A number up to the maximum integer precision available in JavaScript.
• A {hi, lo} or {high, low} pair of 32 bit precision integers.
• A 16 digit hexadecimal string, like 0102030405060708.
• An array of 8 byte values. Ranges are not checked, but coerced.
• An 8 byte buffer.

ThriftRW supports a type annotation for i64 that reifies i64 as an instance of Long, an object that models a 64 bit number as a {high, low} pair of 32 bit numbers and implements methods for common mathematical operators.

typedef i64 (js.type = 'Long') long

Please use the Long package for operating with such numbers. A backward-incompatible release may make this form the default for reading.

With the Long type annotation, ThriftRW performs the following conversions on your behalf.

var Long = require('long');

// From a buffer to a long
var buf = new Buffer("0102030405060708", "hex");
var num = Long.fromBits(buf.readInt32BE(4, true), buf.readInt32BE(0, true));

// From a long to a buffer
var buf = new Buffer(8);
buf.writeUInt32BE(num.high, 0, 4, true);
buf.writeUInt32BE(num.low, 4, 4, true);

Timestamps

The convention for expressing timestamps at Uber is to use an i64 for UTC milliseconds since the UNIX epoch. Previously, some services were found spending an unreasonable amount of time parsing timestamp strings. ThriftRW supports a (js.type = 'Date') annotation on i64.

typedef i64 (js.type = 'Date') timestamp

This will reify timestamps as a JavaScript Date and ThriftRW will accept an ISO-8601 timestamp or milliseconds since the epoch (as returned by Date.now()) in place of a Date.

For TCurl, this means that you can both read and write ISO-8601 for timestamps.

Releated

thriftrw-python is the sister library for Python.

Installation

npm install thriftrw

Tests

npm test

NPM scripts

• npm run add-licence This will add the licence headers.
• npm run cover This runs the tests with code coverage
• npm run lint This will run the linter on your code
• npm test This will run the tests.
• npm run trace This will run your tests in tracing mode.
• npm run travis This is run by travis.CI to run your tests
• npm run view-cover This will show code coverage in a browser

Contributors

• Lei Zhao @leizha
• Kris Kowal @kriskowal
• Andrew de Andrade @malandrew

MIT Licenced

Changelog

[3.0.0] - 2015-11-16

Added

• Introduces Thrift includes. See MIGRATION.md for details regarding changes to the public interface.