protobufjs

What is protobufjs?

The protobufjs npm package provides a comprehensive suite of tools for working with Protocol Buffers (protobuf), a method of serializing structured data. It allows users to encode and decode protobuf messages, generate and work with static code, and handle dynamic message building and parsing.

What are protobufjs's main functionalities?

Loading .proto files

This feature allows users to load .proto files and use the defined protobuf structures within their JavaScript code.

const protobuf = require('protobufjs');
protobuf.load('awesome.proto', function(err, root) {
  if (err) throw err;
  const AwesomeMessage = root.lookupType('awesomepackage.AwesomeMessage');
  // ... use AwesomeMessage
});

Encoding and decoding messages

With protobufjs, users can encode JavaScript objects into binary protobuf format and decode binary messages into JavaScript objects.

const message = AwesomeMessage.create({ awesomeField: 'AwesomeString' });
const buffer = AwesomeMessage.encode(message).finish();
const decodedMessage = AwesomeMessage.decode(buffer);

Reflection and runtime message building

This feature allows users to work with protobuf messages dynamically at runtime using JSON descriptors, without the need for generated static code.

const root = protobuf.Root.fromJSON(jsonDescriptor);
const AwesomeMessage = root.lookupType('awesomepackage.AwesomeMessage');
const errMsg = AwesomeMessage.verify({ awesomeField: 'AwesomeString' });
if (errMsg) throw Error(errMsg);
const message = AwesomeMessage.create({ awesomeField: 'AwesomeString' });

Static code generation

Protobufjs can generate static code from .proto files, which can be used for better performance and type safety.

protobuf.load('awesome.proto', function(err, root) {
  if (err) throw err;
  protobuf.codegen(root, { keepCase: true }, function(err, output) {
    if (err) throw err;
    // output will contain the generated static code
  });
});

Other packages similar to protobufjs

@apollo/protobufjs

This is a fork of the original protobufjs package with some modifications. It is used within the Apollo tooling ecosystem but generally offers similar functionality to protobufjs.

google-protobuf

This is the official Protocol Buffers runtime library for JavaScript. It is provided by Google and offers similar serialization and deserialization capabilities. However, it may not be as feature-rich or flexible as protobufjs in terms of dynamic message handling and may require more setup for code generation.

pbf

Pbf is a fast, lightweight Protocol Buffers implementation in JavaScript. It focuses on performance and is smaller in size compared to protobufjs. However, it might not offer the same level of functionality, especially in terms of reflection and dynamic message building.

README

A protobuf implementation on top of ByteBuffer.js including a .proto parser, reflection, message class building and simple encoding and decoding in plain JavaScript. No compilation step required, works out of the box on .proto files. It's also used for the cross-platform multiplayer component in eSoccer, an HTML5 game developed at University of Applied Sciences Bonn.

Features

• CommonJS compatible
• RequireJS/AMD compatible
• Shim compatible (include the script, then use var ProtoBuf = dcodeIO.ProtoBuf;)
• node.js compatible, also available via npm
• Closure Compiler compatible (fully annotated, externs)
• Fully documented using jsdoc3
• Well tested through nodeunit
• ByteBuffer.js is the only production dependency
• Small footprint (even smaller if you use a noparse build)

Builder

Probably the core component of ProtoBuf.js. Resolves all type references, performs all the necessary checks and returns ready to use classes. Can be created from a .proto file or from a JSON definition. The later does not even require the .proto parser to be included (see).

Install: npm install protobufjs

Using .proto files

Example: tests/complex.proto

var ProtoBuf = require("protobufjs");

var builder = ProtoBuf.protoFromFile("tests/complex.proto");
var Game = builder.build("Game");
var Car = Game.Cars.Car;

// Construct with arguments list in field order:
var car = new Car("Rusty", new Car.Vendor("Iron Inc.", new Car.Vendor.Address("US")), Car.Speed.SUPERFAST);

// OR: Construct with values from an object, implicit message creation (address) and enum values as strings:
var car = new Car({
    "model": "Rusty",
    "vendor": {
        "name": "Iron Inc.",
        "address": {
            "country": "US"
        }
    },
    "speed": "SUPERFAST" // also equivalent to "speed": 2
});

// OR: It's also possible to mix all of this!

// Afterwards, just encode your message:
var buffer = car.encode();

// And send it over the wire:
var socket = ...;
socket.send(buffer.toArrayBuffer());

// OR: Short...
socket.send(car.toArrayBuffer());

Putting multiple .proto files into a common namespace

Example: tests/example1.proto, tests/example2.proto

var builder = ProtoBuf.protoFromFile("tests/example1.proto");
ProtoBuf.protoFromFile("tests/example2.proto", builder);
var root = builder.build();
var test1 = new root.Test1(123);
var test2 = new root.Test2("123");
...

Using JSON without the .proto parser

Example: tests/complex.json

var ProtoBuf = require("protobufjs");

var builder = ProtoBuf.newBuilder();    // Alternatively:
builder.define("Game");                 // var builder = ProtoBuf.newBuilder("Game");
builder.create([
      {
          "name": "Car",
          "fields": [
              {
                  "rule": "required",
                  "type": "string",
                  "name": "model",
                  "id": 1
              },
              ...
          ],
          "messages": [
              {
                  "name": "Vendor",
                  "fields": ...,
              },
              ...
          ],
          "enums": [
              {
                  "name": "Speed",
                  "values": [
                      {
                          "name": "FAST",
                          "id": 1
                      },
                      ...
                  ]
              }
          ]
      }
]);

var Game = builder.build("Game");
var Car = Game.Cars.Car;
... actually the same as above ...

When using JSON only, you can use ProtoBuf.noparse.js / ProtoBuf.noparse.min.js instead, which do NOT include the ProtoBuf.DotProto package for parsing and are therefore even smaller.

Encoder / Decoder

Built into all message classes. Just call YourMessage#encode([buffer]) respectively YourMessage.decode(buffer).

Encoding a message

...
var YourMessage = builder.build("YourMessage");
var myMessage = new YourMessage(...);
var byteBuffer = myMessage.encode();
var buffer = byteBuffer.toArrayBuffer();

// OR: Short...
var buffer = myMessage.toArrayBuffer();

var socket = ...; // E.g. a WebSocket
socket.send(buffer);

Decoding from an ArrayBuffer/ByteBuffer

...
var YourMessage = builder.build("YourMessage");
var buffer = ...; // E.g. a buffer received on a WebSocket
var myMessage = YourMessage.decode(buffer);

Command line utility

It's also possible to transform .proto files into their JSON counterparts or to transform entire namespaces into ready-to-use message classes and enum objects by using the proto2js command line utility.

  Usage: proto2js protoFile [-class[=My.Package]|-commonjs[=My.Package]|-amd[=My.Package]] [-min] [> outFile]

  Options:

    -class[=My.Package]     Creates the classes instead of just a JSON definition.
                            If you do not specify a package, the package
                            declaration from the .proto file is used instead.

    -commonjs[=My.Package]  Creates a CommonJS export instead of just a JSON def.
                            If you do not specify a package, the package
                            declaration from the .proto file is used instead.

    -amd[=My.Package]       Creates an AMD define instead of just a JSON def.
                            If you do not specify a package, the package
                            declaration from the .proto file is used instead.

    -min                    Minifies the output.

So, to create a JSON definition from the tests/complex.proto file, run:

proto2js tests/complex.proto > tests/complex.json

Or to create classes/a CommonJS export/an AMD define for the entire Game namespace, run:

proto2js tests/complex.proto -class=Game > tests/complex.js

proto2js tests/complex.proto -commonjs=Game > tests/complex.js
# ^ The resulting code will implicitly call require("protobufjs");

proto2js tests/complex.proto -amd=Game.Cars > tests/complex.js
# ^ The resulting code will implicitly use ["ProtoBuf"] as its
#   declared dependencies and "Game/Cars" as the module name

Parser

Only available in the full build (i.e. not in "noparse" builds). Compliant with the protobuf parser to the following extend:

• Required, optional, repeated and packed repeated fields:

  message Test {
      required int32 a = 1;
      optional int32 b = 2 [default=100];
      repeated int32 c = 3;
      repeated int32 c = 4 [packed=true];
  }
  

• Data types: int32, uint32, sint32, fixed32, sfixed32, int64, uint64, sint64, fixed64, sfixed64, bool, enum, string, bytes, messages, embedded messages, float, double

  message Test {
      required int32 a = 1; // Varint encoded
      required uint32 b = 2; // Varint encoded
      required sint32 c = 3; // Varint zigzag encoded
      required bool d = 4; // Varint encoded
    
      enum Priority {
          LOW = 1;
          MEDIUM = 2;
          HIGH = 3;
      }
    
      optional Priority e = 5 [default=MEDIUM]; // Varint encoded
      required string f = 6; // Varint length delimited
      required bytes g = 7; // Varint length delimited
      required Embedded h = 8; // Varint length delimited
    
      message Embedded {
          repeated int32 a = 1; // Multiple tags
          repeated int32 b = 2 [packed=true]; // One tag, length delimited
          
          required sfixed32 c = 3; // Fixed 4 bytes int32
          required fixed32 d = 4; // Fixed 4 bytes uint32
          required float e = 5; // Fixed 4 bytes
          required double f = 6; // Fixed 8 bytes
          
          // Through ByteBuffer.js loaded with Long.js:
          optional int64 g = 7; // Varint encoded
          optional uint64 h = 8; // Varint encoded
          optional sint64 i = 9; // Varint zigzag encoded
          required sfixed64 j = 10; // Fixed 8 bytes int64
          required fixed64 k = 11; // Fixed 8 bytes uint64
      }
  }
  

• Packages

  package My.Game;
  
  message Test {
      ...
  }
  
  message Test2 {
      required My.Game.Test test = 1;
  }
  

• Qualified and fully qualified name resolving:

  package My.Game;
  
  message Test {
      ...
      
      enum Priority {
          LOW = 1;
          MEDIUM = 2;
          HIGH = 3;
      }
  }
  
  message Test2 {
      required .My.Game.Test.Priority priority_fqn = 1 [default=LOW];
      required Test.Priority priority_qn = 2 [default=MEDIUM];
  }
  

• Options on all levels:

  option toplevel_1 = 10;
  option toplevel_2 = "Hello!";
  
  message Test {
      option inmessage = "World!";
      optional int32 somenumber = 1 [default=123]; // Actually the only one used
  }
  

  Paranthesis around option names, like option (inmessage) = ... are ignored for compatibility with custom options. Accessing arbitrary options on namespace objects emitted by a builder is also possible (requires an ECMAScript 5 / Javascript 1.8.5 compatible environment, e.g. node.js or any modern browser):

  ...
  var root = builder.build();
  console.log(root.$options); // { 'toplevel_1': 10, 'toplevel_2': 'Hello!' }
  console.log(root.Test.$options); // { 'inmessage': 'World!' }
  

• Imports:

  package My; // Not used by the following imports
  import public "example1.proto"; // The "public" keyword is ignored
  import "example2.proto";
  
  message Test2 {
      required Test1 test1 = 0; // Unique name
      required .Test2 test2 = 1; // Duplicate name resolved absolutely
  }
  

  See also: example1.proto, example2.proto

  var builder = ProtoBuf.protoFromFile("tests/imports.proto");
  var root = builder.build();
  // root now contains:
  // - .Test1
  // - .Test2
  // - .My.Test2 using .Test1 for field 'test1' and .Test2 for field 'test2'
  

  NOTE: Importing .proto files is available in the full build (not in the noparse build) only. Importing the .json counterpart of a .proto file (e.g. generated by proto2js) is supported regardless of the build you are using. Imports are also always fetched synchronously, so it's a good idea to package everything into a single file when using it in the browser.

• Extensions:

  extend something {
      ...
  }
  

  import "google/protobuf/descriptor.proto";
  

  These statements are actually simply ignored to make existing definitions compile fine. Low-level bootstrapping is not required by ProtoBuf.js.

Not (yet) supported

• Extensions (what for?) and services (you roll your own, don't you?). If you need anything of this, please open an issue and explain your exact requirements. It's just that I have no idea how to benefit from that and therefore I am not sure how to design it.

Calling the parser on your own

var ProtoBuf = require("protobufjs"),
    fs = require("fs"),
    util = require("util");

var parser = new ProtoBuf.DotProto.Parser(fs.readFileSync("tests/complex.proto"));
var ast = parser.parse();
console.log(util.inspect(ast, false, null, true));

CommonJS, AMD and browser (shim)

node.js / CommonJS

npm install protobufjs

var ProtoBuf = require("protobufjs");
...

RequireJS / AMD

Requires ByteBuffer.js. Optionally depends on Long.js for long (int64) support. If you do not require long support, you can skip the Long.js config. Require.js example:

require.config({
    ...
    "paths": {
        "Long": "/path/to/Long.js",
        "ByteBuffer": "/path/to/ByteBuffer.js",
        "ProtoBuf": "/path/to/ProtoBuf.js"
    },
    ...
});
require(["ProtoBuf"], function(ProtoBuf) {
    ...
});

Or as a module dependency:

define("MyModule", ["ProtoBuf"], function(ProtoBuf) {
    ...
});

Browser (shim)

Requires ByteBuffer.js. Optionally depends on Long.js for long (int64) support. If you do not require long support, you can skip the Long.js include.

<script src="//raw.github.com/dcodeIO/Long.js/master/Long.min.js"></script>
<script src="//raw.github.com/dcodeIO/ByteBuffer.js/master/ByteBuffer.min.js"></script>
<script src="//raw.github.com/dcodeIO/ProtoBuf.js/master/ProtoBuf.min.js"></script>

var ProtoBuf = dcodeIO.ProtoBuf;
...

On long (int64) support

Full 64bit support is available since 0.10.0 and requires ByteBuffer.js >=1.2.3 with Long.js >=1.1.2.

• See also

On endianess before 0.11

Till 0.10, ProtoBuf.js mistakenly used big endian byte order when en-/decoding non-varint values as reported by bertdouglas. This has been corrected in 0.11.0. However, versions before 0.11 are now to consider incompatible with later versions when using non-varint values, so please upgrade to the latest version to ensure compatibility with the official protobuf specification.

Downloads

• ZIP-Archive
• Tarball

Documentation

• View documentation

Tests (& Examples)

• View source
• View node report
• View browser report

Contributors

Frank Xu

License

Apache License, Version 2.0 - http://www.apache.org/licenses/LICENSE-2.0.html