protobufjs

What is protobufjs?

The protobufjs npm package provides a comprehensive suite of tools for working with Protocol Buffers (protobuf), a method of serializing structured data. It allows users to encode and decode protobuf messages, generate and work with static code, and handle dynamic message building and parsing.

What are protobufjs's main functionalities?

Loading .proto files

This feature allows users to load .proto files and use the defined protobuf structures within their JavaScript code.

const protobuf = require('protobufjs');
protobuf.load('awesome.proto', function(err, root) {
  if (err) throw err;
  const AwesomeMessage = root.lookupType('awesomepackage.AwesomeMessage');
  // ... use AwesomeMessage
});

Encoding and decoding messages

With protobufjs, users can encode JavaScript objects into binary protobuf format and decode binary messages into JavaScript objects.

const message = AwesomeMessage.create({ awesomeField: 'AwesomeString' });
const buffer = AwesomeMessage.encode(message).finish();
const decodedMessage = AwesomeMessage.decode(buffer);

Reflection and runtime message building

This feature allows users to work with protobuf messages dynamically at runtime using JSON descriptors, without the need for generated static code.

const root = protobuf.Root.fromJSON(jsonDescriptor);
const AwesomeMessage = root.lookupType('awesomepackage.AwesomeMessage');
const errMsg = AwesomeMessage.verify({ awesomeField: 'AwesomeString' });
if (errMsg) throw Error(errMsg);
const message = AwesomeMessage.create({ awesomeField: 'AwesomeString' });

Static code generation

Protobufjs can generate static code from .proto files, which can be used for better performance and type safety.

protobuf.load('awesome.proto', function(err, root) {
  if (err) throw err;
  protobuf.codegen(root, { keepCase: true }, function(err, output) {
    if (err) throw err;
    // output will contain the generated static code
  });
});

Other packages similar to protobufjs

@apollo/protobufjs

This is a fork of the original protobufjs package with some modifications. It is used within the Apollo tooling ecosystem but generally offers similar functionality to protobufjs.

google-protobuf

This is the official Protocol Buffers runtime library for JavaScript. It is provided by Google and offers similar serialization and deserialization capabilities. However, it may not be as feature-rich or flexible as protobufjs in terms of dynamic message handling and may require more setup for code generation.

pbf

Pbf is a fast, lightweight Protocol Buffers implementation in JavaScript. It focuses on performance and is smaller in size compared to protobufjs. However, it might not offer the same level of functionality, especially in terms of reflection and dynamic message building.

README

A protobuf implementation on top of ByteBuffer.js including a .proto parser, reflection, message class building and simple encoding and decoding in plain JavaScript. No compilation step required, works out of the box on .proto files. It's also used for the cross-platform multiplayer component in eSoccer, an HTML5 game developed at University of Applied Sciences Bonn.

Features

• CommonJS compatible
• RequireJS/AMD compatible
• Shim compatible (include the script, then use var ProtoBuf = dcodeIO.ProtoBuf;)
• node.js compatible, also available via npm
• Closure Compiler compatible (fully annotated, externs)
• Fully documented using jsdoc3
• Well tested through nodeunit
• ByteBuffer.js is the only production dependency
• Small footprint (even smaller if you use a noparse build)

Builder

Probably the core component of ProtoBuf.js. Resolves all type references, performs all the necessary checks and returns ready to use classes. Can be created from a .proto file or from a JSON definition. The later does not even require the .proto parser to be included (see).

Install: npm install protobufjs

Using .proto files

Example: tests/complex.proto

var ProtoBuf = require("protobufjs");

var builder = ProtoBuf.protoFromFile("tests/complex.proto");
var Game = builder.build("Game");
var Car = Game.Cars.Car;

// Construct with arguments list in field order:
var car = new Car("Rusty", new Car.Vendor("Iron Inc.", new Car.Vendor.Address("US")), Car.Speed.SUPERFAST);

// OR: Construct with values from an object, implicit message creation (address) and enum values as strings:
var car = new Car({
    "model": "Rusty",
    "vendor": {
        "name": "Iron Inc.",
        "address": {
            "country": "US"
        }
    },
    "speed": "SUPERFAST" // also equivalent to "speed": 2
});

// OR: It's also possible to mix all of this!

// Afterwards, just encode your message:
var buffer = car.encode();

// And send it over the wire:
var socket = ...;
socket.send(buffer.toArrayBuffer());

// OR: Short...
socket.send(car.toArrayBuffer());

Putting multiple .proto files into a common namespace

Example: tests/example1.proto, tests/example2.proto

var builder = ProtoBuf.protoFromFile("tests/example1.proto");
ProtoBuf.protoFromFile("tests/example2.proto", builder);
var root = builder.build();
var test1 = new root.Test1(123);
var test2 = new root.Test2("123");
...

Using JSON without the .proto parser

Example: tests/complex.json

var ProtoBuf = require("protobufjs");

var builder = ProtoBuf.newBuilder();    // Alternatively:
builder.define("Game");                 // var builder = ProtoBuf.newBuilder("Game");
builder.create([
      {
          "name": "Car",
          "fields": [
              {
                  "rule": "required",
                  "type": "string",
                  "name": "model",
                  "id": 1
              },
              ...
          ],
          "messages": [
              {
                  "name": "Vendor",
                  "fields": ...,
              },
              ...
          ],
          "enums": [
              {
                  "name": "Speed",
                  "values": [
                      {
                          "name": "FAST",
                          "id": 1
                      },
                      ...
                  ]
              }
          ]
      }
]);

var Game = builder.build("Game");
var Car = Game.Cars.Car;
... actually the same as above ...

When using JSON only, you can use ProtoBuf.noparse.js / ProtoBuf.noparse.min.js instead, which do NOT include the ProtoBuf.DotProto package for parsing and are therefore even smaller.

Command line utility

It's also possible to transform .proto files into their JSON counterparts or to transform entire namespaces into ready-to-use message classes and enum objects by using the proto2js command line utility.

  Usage: proto2js protoFile [-class[=My.Package]|-commonjs[=My.Package]|-amd[=My.Package]] [-min] [> outFile]

  Options:

    -class[=My.Package]     Creates the classes instead of just a JSON definition.
                            If you do not specify a package, the package
                            declaration from the .proto file is used instead.

    -commonjs[=My.Package]  Creates a CommonJS export instead of just a JSON def.
                            If you do not specify a package, the package
                            declaration from the .proto file is used instead.

    -amd[=My.Package]       Creates an AMD define instead of just a JSON def.
                            If you do not specify a package, the package
                            declaration from the .proto file is used instead.

    -min                    Minifies the output.

So, to create a JSON definition from the tests/complex.proto file, run:

proto2js tests/complex.proto > tests/complex.json

Or to create classes/a CommonJS export/an AMD define for the entire Game namespace, run:

proto2js tests/complex.proto -class=Game > tests/complex.js

proto2js tests/complex.proto -commonjs=Game > tests/complex.js
# ^ The resulting code will implicitly call require("protobufjs");

proto2js tests/complex.proto -amd=Game.Cars > tests/complex.js
# ^ The resulting code will implicitly use ["ProtoBuf"] as its
#   declared dependencies and "Game/Cars" as the module name

Parser

Only available in the full build (i.e. not in "noparse" builds). Compliant with the protobuf parser to the following extend:

• Required, optional, repeated and packed repeated fields:

  message Test {
      required int32 a = 1;
      optional int32 b = 2 [default=100];
      repeated int32 c = 3;
      repeated int32 c = 4 [packed=true];
  }
  

• Data types: int32, uint32, sint32, bool, enum, string, bytes, messages, embedded messages, fixed32, sfixed32, float, double, fixed64, sfixed64:

  message Test {
      required int32 a = 1; // Varint encoded
      required uint32 b = 2; // Varint encoded
      required sint32 c = 3; // Varint zigzag encoded
      required bool d = 4; // Varint encoded
    
      enum Priority {
          LOW = 1;
          MEDIUM = 2;
          HIGH = 3;
      }
    
      optional Priority e = 5 [default=MEDIUM]; // Varint encoded
      required string f = 6; // Varint length delimited
      required bytes g = 7; // Varint length delimited
      required Embedded h = 8; // Varint length delimited
    
      message Embedded {
          repeated int32 a = 1; // Multiple tags
          repeated int32 b = 2 [packed=true]; // One tag, length delimited
          required fixed32 c = 3; // Fixed 4 bytes
          required sfixed32 d = 4; // Fixed 4 bytes zigzag encoded
          required float e = 5; // Fixed 4 bytes
          required double f = 6; // Fixed 8 bytes
          required sfixed64 g = 7; // Fixed 8 bytes (int64 through Long.js)
          required fixed64 h = 8; // Fixed 8 bytes (uint64 through Long.js)
      }
  }
  

• Packages

  package My.Game;
  
  message Test {
      ...
  }
  
  message Test2 {
      required My.Game.Test test = 1;
  }
  

• Qualified and fully qualified name resolving:

  package My.Game;
  
  message Test {
      ...
      
      enum Priority {
          LOW = 1;
          MEDIUM = 2;
          HIGH = 3;
      }
  }
  
  message Test2 {
      required .My.Game.Test.Priority priority_fqn = 1 [default=LOW];
      required Test.Priority priority_qn = 2 [default=MEDIUM];
  }
  

• Options on all levels:

  option toplevel_1 = 10;
  option toplevel_2 = "Hello!";
  
  message Test {
      option inmessage = "World!";
      optional int32 somenumber = 1 [default=123]; // Actually the only one used
  }
  

Not (yet) supported

• Extensions (what for?), imports (put everything into one builder instead by calling Builder#define(package) and Builder#create([messages]) multiple times) and services (you roll your own, don't you?).

However, if you need anything of the above, please drop me a note how you'd like to see it implemented. It's just that I have no idea how to benefit from that and therefore I am not sure how to design it.

Calling the parser on your own

var ProtoBuf = require("protobufjs"),
    fs = require("fs"),
    util = require("util");

var parser = new ProtoBuf.DotProto.Parser(fs.readFileSync("tests/complex.proto"));
var ast = parser.parse();
console.log(util.inspect(ast, false, null, true));

Encoder / Decoder

Built into all message classes. Just call YourMessage#encode([buffer]) respectively YourMessage.decode(buffer).

Encoding a message

...
var YourMessage = builder.build("YourMessage");
var myMessage = new YourMessage(...);
var byteBuffer = myMessage.encode();
var buffer = byteBuffer.toArrayBuffer();

// OR: Short...
var buffer = myMessage.toArrayBuffer();

var socket = ...; // E.g. a WebSocket
socket.send(buffer);

Decoding from an ArrayBuffer/ByteBuffer

...
var YourMessage = builder.build("YourMessage");
var buffer = ...; // E.g. a buffer received on a WebSocket
var myMessage = YourMessage.decode(buffer);

CommonJS, AMD and browser (shim)

node.js / CommonJS

npm install protobufjs

var ProtoBuf = require("protobufjs");
...

RequireJS / AMD

Requires ByteBuffer.js. Optionally depends on Long.js for long (int64) support. If you do not require long support, you can skip the Long.js config. Require.js example:

require.config({
    ...
    "paths": {
        "Long": "/path/to/Long.js",
        "ByteBuffer": "/path/to/ByteBuffer.js",
        "ProtoBuf": "/path/to/ProtoBuf.js"
    },
    ...
});
require(["ProtoBuf"], function(ProtoBuf) {
    ...
});

Or as a module dependency:

define("MyModule", ["ProtoBuf"], function(ProtoBuf) {
    ...
});

Browser (shim)

Requires ByteBuffer.js. Optionally depends on Long.js for long (int64) support. If you do not require long support, you can skip the Long.js include.

<script src="//raw.github.com/dcodeIO/Long.js/master/Long.min.js"></script>
<script src="//raw.github.com/dcodeIO/ByteBuffer.js/master/ByteBuffer.min.js"></script>
<script src="//raw.github.com/dcodeIO/ProtoBuf.js/master/ProtoBuf.min.js"></script>

var ProtoBuf = dcodeIO.ProtoBuf;
...

Downloads

• ZIP-Archive
• Tarball

Documentation

• View documentation

Tests (& Examples)

• View source
• View report

License

Apache License, Version 2.0 - http://www.apache.org/licenses/LICENSE-2.0.html