thriftrw
Encodes and decodes Thrift binary protocol and JavaScript object models
declared in a Thrift IDL.
This is an alternative approach to using code generated from Thrift IDL source
files with the thrift
compiler.
ThriftRW supports encoding and decoding the protocol with or without Thrift IDL
sources.
Without sources, it is still possible to read and write the Thrift binary
protocol in all of its structure using numbered fields for structs and not
discerning the nuanced differences between binary and string, list and set, or
number and enum.
With a Thrift IDL, ThriftRW is able to perform certain optimizations.
ThriftRW constructs model instances directly from the contents of a buffer
instead of creating an intermediate anonymous structural model.
As a consequence, it is able to quickly skip any unrecognized fields.
ThriftRW can also use the same struct constructor for every instance of a
struct, which should yield run-time performance benefits for V8 due to hidden
classes (not yet verified).
The scope of this project may eventually cover alternate Thrift binary
encodings including the compact binary protocol.
This project makes extensive use of bufrw for reading and writing binary
protocols, a component shared by tchannel-node, with which this library
works in concert.
Example
ThriftRW provides a Thrift constructor that models all of the services,
functions, and types expressed in a ThriftIDL source file.
var fs = require('fs');
var path = require('path');
var Thrift = require('thriftrw').Thrift;
var source = fs.readFileSync(fs.path(__dirname, 'meta.thrift'), 'ascii');
var thrift = new Thrift({source: source, strict: true});
Consider meta.thrift
struct HealthStatus {
1: required bool ok
2: optional string message
}
service Meta {
HealthStatus health()
string thriftIDL()
}
The most common usage of a Thrift instance is to get the argument and result
types of a function and use those types to read to or write from a buffer.
The getType
and getTypeResult
functions retrieve such models based on their name.
For example, the arguments struct for the health endpoint would be Meta::health_args
and its result struct would be Meta::health_result
.
var MetaHealthArgs = thrift.getType('Meta::health_args');
var MetaHealthResult = thrift.getType('Meta::health_result');
The getType
method will return the struct or throw an exception if one does not exist.
This is the appropriate method if getType
should throw an error due to a
programmer's mistake, where the Thrift source in question is checked into the same project
and the method name is a literal.
However, exceptions being an undesirable outcome for bad data from a user,
getTypeResult
returns a result object with either err
or value
properties set.
var res = thrift.getTypeResult(methodName + '_args');
if (res.err) {
return callback(res.err);
}
var ArgsStruct = res.value;
The struct can be written or read to or from a buffer using the struct's rw
instance.
A RW (reader/writer) implements byteLength(value)
, writeInto(value, buffer, offset)
, readFrom(buffer, offset)
, each of which return a result object as
specified by bufrw.
The value may be JSON, a POJO (plain old JavaScript object), or any instances
of the Thrift model, like new thrift.Health({ok: true})
.
A struct can also be encoded or decoded using the ThriftStruct's own
toBuffer(value)
, fromBuffer(buffer)
, toBufferResult(value)
, and
fromBufferResult(buffer)
methods.
Those with Result
in their name return Result
instances instead of throwing
or returning the buffer or value.
TChannelASThrift employs this interface on your behalf, leaving the
application author to take arguments and call back with a result when using
the Thrift argument scheme and a given Thrift IDL.
Without Thrift IDL
ThriftRW provides T
prefixed types for encoding and decoding the wire
protocol without Thrift IDL.
This is useful for applications like [TCap] that do not necessarily have a
Thrift IDL file on hand to make sense of structs by field numbers alone.
The following example illustrates reading and writing a struct.
ThriftRW exports TStruct
, TList
, and TMap
.
TStructs serve for arguments, results, and exceptions.
TLists also serve for sets.
var thriftrw = require("thriftrw");
var bufrw = require('bufrw');
var struct = new thriftrw.TStruct();
struct.fields.push(
new thriftrw.TField(thriftrw.TYPE.STRING, 1, new Buffer('hello')
);
var buf = bufrw.toBuffer(thriftrw.TStructRW, struct);
console.log('created a binary buffer of thrift encoded struct', buf);
var struct2 = bufrw.fromBuffer(thriftrw.TStructRW, buf);
console.log('created a TStruct from a binary buffer', struct2);
Thrift Model
Thrift has internal models for modules, services, types, and values.
These models are indexed by name on each Thrift module in the models
object.
Each of these models has a "surface", which is a constructor for types, values
for constants, an index of functions for services, and the model itself for
modules.
Modules expose their names on their own object only if they start with a
non-lower-case character, to avoid name collisions with other properties and
methods, but are always also available through the index for their type class,
like thrift.consts.PI
.
The Meta service is simply thrift.Meta
.
The object is an object mapping functions by name, so thrift.Meta.health
is
the interface for accessing the args
and result
structs for the
Meta::health
function.
var args = new thrift.Meta.health.Arguments({ok: true});
var result = new thrift.Meta.health.Result({success: null});
The Thrift instance also has a services
property containing the actual
ThriftService instance indexed by service name.
ThriftService instances are not expected to be useful outside the process of
compiling and linking the Thrift IDL, but you can use the services object to
check for the existence of a service by its name.
Structs
ThriftStruct models can be constructed with an optional object of specified
properties.
ThriftRW exposes the constructor for a struct declaration by name on the thrift
instance, as in thrift.Health
for the meta.thrift example.
var HealthStruct = thrift.Health;
var result = new ResultStruct({
success: new HealthStruct({ok: true })
})
Unspecified properties will default to null or an instance of the default value
specified in the Thrift IDL.
Nested structs can be expressed with JSON or POJO equivalents.
The constructors perform no validation.
Invalid objects are revealed only in the attempt to write one to a buffer.
var result = new ResultStruct({success: {ok: true}});
Each constructor has a rw
property that reveals the reader/writer instance.
RW objects imlement byteLength(value)
, readFrom(buffer, offset)
, and
writeInto(value, buffer, offset)
.
The value may be any object of the requisite shape, though using the given
constructors increases the probability V8 optimization.
Each constructor also hosts toBuffer(value)
, fromBuffer(buffer)
,
toBufferResult(value)
, and fromBufferResult(buffer)
.
Structs are indexed by name on the thrift.structs
object and aliased on the
thrift object if their name does not start with a lower-case letter.
Exceptions
ThriftException extends ThriftStruct.
Exceptions are modeled as structs on the wire, and are modeled as JavaScript
exceptions instead of regular objects.
As such they have a stack trace.
ThriftRW exposes the exception constructor by name on the thrift instance.
exception Pebcak {
1: required string message
2: required string keyboardName
3: required string chairName
}
var error = new thrift.Pebcak({
message: 'Problem exists between chair and keyboard',
chairName: 'Hengroen',
keyboardName: 'Excalibur'
});
Exceptions are indexed by name on the thrift.exceptions
object and aliased on
the thrift object if their name does not start with a lower-case letter.
Unions
ThriftUnion also extends ThriftStruct.
Unions are alike to structs except that fields must not be marked as optional
or required
, cannot have a default value, and exactly one must be defined on
an instance.
As with other types, validation of a union only occurs when it is read or
written.
union CoinToss {
1: Obverse heads
2: Reverse tails
}
struct Obverse {
1: required string portrait;
2: required i32 year;
}
struct Reverse {
1: required string structure;
2: optional string motto;
}
var coinToss = new thrift.CoinToss({
head: thrift.Obverse({
portrait: 'TK',
year: 2010
})
})
Unions are indexed by name on the thrift.unions
object and aliased on the
thrift object if their name does not start with a lower-case letter.
Enums
Thrift enumerations are surfaced as an object mapping names to strings on
the thrift instance.
enum CoinToss {
tails = 0
heads = 1
}
var result = Math.random() < 0.5 ?
thrift.CoinToss.heads :
thrift.CoinToss.tails;
ThriftRW hides the detail of which numeric value an enumeration will have on
the wire and allows an enumeration to be represented in JSON by name.
This is a deliberate departure from the norms for Thrift in other languages
to take advantage of the dynamic nature of JavaScript, permit duck typing,
enforce loose coupling between application code and wire protocol, and make
Thrift expressible as JSON on the command line with TCurl
Enums are indexed by name on the thrift.unions
object and aliased on the
thrift object if their name does not start with a lower-case letter.
Consts
Thrift constants are surfaced as properties of the thrift instance.
const PI = 3
const TAU = 6
var PI = thrift.PI;
var TAU = thrift.TAU;
Consts are indexed by name on the thrift.consts
object and aliased on the
thrift object if their name does not start with a lower-case letter.
Enhancements and Departures from Thrift proper
ThriftRW operates in a "strict" mode by default that imposes additional
constraints on Thrift to ensure cross language interoperability between
JavaScript, Python, Go, and Java, the four Thrift bindings favored at Uber.
In strict mode, structs must explicitly declare every field to be either
required, optional, or have a default value.
ThriftRW supports forward references and self-referential types (like trees)
without special demarcation.
ThriftRW also supports forward references in default values and constants.
Types and Annotations
ThriftRW respects certain annotations specific to the treatment of various
types in JavaScript.
set
There is an emerging standard for the representation of sets in JavaScript.
Until that standard becomes pervasive, sets are traditionally represented
either as an array with externally enforced constraints, or as long as the values
are scalar like numbers and strings, as an Object with values for keys.
By default, ThriftRW encodes and decodes sets as arrays.
ThriftRW does nothing to enforce uniqueness.
Additionally, the js.type
annotation can be set to "object" if the value type
is a string, i16, or i32.
ThriftRW will ensure that the keys are coerced to and from Number for numeric
value types.
typedef set<string> (js.type = 'object') StringSet
typedef set<i32> (js.type = 'object') I32Set
map
There is also an emerging standard for the representation of maps in JavaScript.
However, until support is more pervasive, ThriftRW supports representing maps
as either an object with scalar key types or an entries array for all others.
Maps are represented as objects by default. With the js.type
annotation set
to "entries", the map will be encoded and decoded with an array of [key, value]
duples.
typedef map<string, string> Dictionary
typedef map<Struct, Struct> (js.type = 'entries') StructsByStructs
i64
JavaScript numbers lack sufficient precision to represent all possible 64 bit
integers.
ThriftRW decodes 64 bit integers into a Buffer by default, and can coerce
various types down to i64 for purposes of expressing them as JSON or plain
JavaScript objects.
- A number up to the maximum integer precision available in JavaScript.
- A
{hi, lo}
or {high, low}
pair of 32 bit precision integers. - A 16 digit hexadecimal string, like
0102030405060708
. - An array of 8 byte values. Ranges are not checked, but coerced.
- An 8 byte buffer.
ThriftRW supports a type annotation for i64 that reifies i64 as an instance of
Long, an object that models a 64 bit number as a {high, low}
pair of 32
bit numbers and implements methods for common mathematical operators.
typedef i64 (js.type = 'Long') long
Please use the Long package for operating with such numbers.
A backward-incompatible release may make this form the default for reading.
With the Long type annotation, ThriftRW performs the following conversions on
your behalf.
var Long = require('long');
var buf = new Buffer("0102030405060708", "hex");
var num = Long.fromBits(buf.readInt32BE(4, true), buf.readInt32BE(0, true));
var buf = new Buffer(8);
buf.writeUInt32BE(num.high, 0, 4, true);
buf.writeUInt32BE(num.low, 4, 4, true);
Timestamps
The convention for expressing timestamps at Uber is to use an i64 for
UTC milliseconds since the UNIX epoch.
Previously, some services were found spending an unreasonable amount of time
parsing timestamp strings.
ThriftRW supports a (js.type = 'Date')
annotation on i64.
typedef i64 (js.type = 'Date') timestamp
This will reify timestamps as a JavaScript Date and ThriftRW will accept an
ISO-8601 timestamp or milliseconds since the epoch (as returned by
Date.now()
) in place of a Date.
For TCurl, this means that you can both read and write ISO-8601 for
timestamps.
Releated
thriftrw-python is the sister library for Python.
Installation
npm install thriftrw
Tests
npm test
NPM scripts
npm run add-licence
This will add the licence headers.npm run cover
This runs the tests with code coveragenpm run lint
This will run the linter on your codenpm test
This will run the tests.npm run trace
This will run your tests in tracing mode.npm run travis
This is run by travis.CI to run your testsnpm run view-cover
This will show code coverage in a browser
Contributors
- Lei Zhao @leizha
- Kris Kowal @kriskowal
- Andrew de Andrade @malandrew
MIT Licenced