Security News
PyPI’s New Archival Feature Closes a Major Security Gap
PyPI now allows maintainers to archive projects, improving security and helping users make informed decisions about their dependencies.
protobufjs
Advanced tools
The protobufjs npm package provides a comprehensive suite of tools for working with Protocol Buffers (protobuf), a method of serializing structured data. It allows users to encode and decode protobuf messages, generate and work with static code, and handle dynamic message building and parsing.
Loading .proto files
This feature allows users to load .proto files and use the defined protobuf structures within their JavaScript code.
const protobuf = require('protobufjs');
protobuf.load('awesome.proto', function(err, root) {
if (err) throw err;
const AwesomeMessage = root.lookupType('awesomepackage.AwesomeMessage');
// ... use AwesomeMessage
});
Encoding and decoding messages
With protobufjs, users can encode JavaScript objects into binary protobuf format and decode binary messages into JavaScript objects.
const message = AwesomeMessage.create({ awesomeField: 'AwesomeString' });
const buffer = AwesomeMessage.encode(message).finish();
const decodedMessage = AwesomeMessage.decode(buffer);
Reflection and runtime message building
This feature allows users to work with protobuf messages dynamically at runtime using JSON descriptors, without the need for generated static code.
const root = protobuf.Root.fromJSON(jsonDescriptor);
const AwesomeMessage = root.lookupType('awesomepackage.AwesomeMessage');
const errMsg = AwesomeMessage.verify({ awesomeField: 'AwesomeString' });
if (errMsg) throw Error(errMsg);
const message = AwesomeMessage.create({ awesomeField: 'AwesomeString' });
Static code generation
Protobufjs can generate static code from .proto files, which can be used for better performance and type safety.
protobuf.load('awesome.proto', function(err, root) {
if (err) throw err;
protobuf.codegen(root, { keepCase: true }, function(err, output) {
if (err) throw err;
// output will contain the generated static code
});
});
This is a fork of the original protobufjs package with some modifications. It is used within the Apollo tooling ecosystem but generally offers similar functionality to protobufjs.
This is the official Protocol Buffers runtime library for JavaScript. It is provided by Google and offers similar serialization and deserialization capabilities. However, it may not be as feature-rich or flexible as protobufjs in terms of dynamic message handling and may require more setup for code generation.
Pbf is a fast, lightweight Protocol Buffers implementation in JavaScript. It focuses on performance and is smaller in size compared to protobufjs. However, it might not offer the same level of functionality, especially in terms of reflection and dynamic message building.
Protocol Buffers are a language-neutral, platform-neutral, extensible way of serializing structured data for use in communications protocols, data storage, and more, originally designed at Google (see).
protobuf.js is a pure JavaScript implementation with TypeScript support for node and the browser. It's super easy to use, blazingly fast and works out of the box on .proto files!
Recommended read: Changes in protobuf.js 6.0
Usage
How to include protobuf.js in your project.
Examples
A few examples to get you started.
Documentation
A list of available documentation resources.
Command line
How to use the command line utility.
Performance
A few internals and a benchmark on performance.
Compatibility
Notes on compatibility regarding browsers and optional libraries.
Building
How to build the library and its components yourself.
$> npm install protobufjs [--save --save-prefix=~]
var protobuf = require("protobufjs");
Development:
<script src="//cdn.rawgit.com/dcodeIO/protobuf.js/6.X.X/dist/protobuf.js"></script>
Production:
<script src="//cdn.rawgit.com/dcodeIO/protobuf.js/6.X.X/dist/protobuf.min.js"></script>
NOTE: Remember to replace the version tag with the exact release your project depends upon.
Or download the library.
The protobuf
namespace will always be available globally / also supports AMD loaders.
// awesome.proto
package awesomepackage;
syntax = "proto3";
message AwesomeMessage {
string awesome_field = 1; // becomes awesomeField
}
protobuf.load("awesome.proto", function(err, root) {
if (err) throw err;
// Obtain a message type
var AwesomeMessage = root.lookup("awesomepackage.AwesomeMessage");
// Create a new message
var message = AwesomeMessage.create({ awesomeField: "AwesomeString" });
// Encode a message
var buffer = AwesomeMessage.encode(message).finish();
// ... do something with buffer
// Or, encode a plain object
var buffer = AwesomeMessage.encode({ awesomeField: "AwesomeString" }).finish();
// ... do something with buffer
// Decode a buffer
var message = AwesomeMessage.decode(buffer);
// ... do something with message
// If your application uses length-delimited buffers, there is also encodeDelimited and decodeDelimited.
});
You can also use promises by omitting the callback:
protobuf.load("awesome.proto")
.then(function(root) {
...
});
...
var Root = protobuf.Root,
Type = protobuf.Type,
Field = protobuf.Field;
var AwesomeMessage = new Type("AwesomeMessage").add(new Field("awesomeField", 1, "string"));
var root = new Root().define("awesomepackage").add(AwesomeMessage);
// Continue at "Create a new message" above
...
...
function AwesomeMessage(properties) {
protobuf.Message.call(this, properties);
}
protobuf.Class.create(root.lookup("awesomepackage.AwesomeMessage") /* or use reflection */, AwesomeMessage);
var message = new AwesomeMessage({ awesomeField: "AwesomeString" });
// Continue at "Encode a message" above
Custom classes are automatically populated with static encode
, encodeDelimited
, decode
, decodeDelimited
and verify
methods and reference their reflected type via the $type
property. Note that there are no methods (just $type
) on instances by default as method names might conflict with field names.
While only useful for the adventurous cherishing an aversion to generated static code, it's also possible to use the Reader/Writer interface directly depending just on the minimal runtime (basic example).
Easy ways to obtain example code snippets are either setting protobuf.util.codegen.verbose = true
while watching the magic as it happens, or simply inspecting generated static code.
// greeter.proto
syntax = "proto3";
service Greeter {
rpc SayHello (HelloRequest) returns (HelloReply) {}
}
message HelloRequest {
string name = 1;
}
message HelloReply {
string message = 1;
}
...
var Greeter = root.lookup("Greeter");
var greeter = Greeter.create(rpcImpl, false, false); // rpcImpl (see below), requestDelimited?, responseDelimited?
greeter.sayHello({ name: 'you' }, function(err, response) {
console.log('Greeting:', response.message);
});
To make this work, all you have to do is provide an rpcImpl
, which is an asynchronous function that takes the reflected service method, the binary HelloRequest and a node-style callback as its parameters. For example:
function rpcImpl(method, requestData, callback) {
// perform the request using an HTTP request or a WebSocket for example
var responseData = ...;
// and call the callback with the binary response afterwards:
callback(null, responseData);
}
There is also an example for streaming RPC.
import * as protobuf from "protobufjs";
import * as Long from "long"; // optional
...
See also: Generating your own TypeScript definitions
Additional configuration might be necessary when not utilizing node, i.e. reference protobuf.js.d.ts and long.js.d.ts.
The pbjs
command line utility can be used to bundle and translate between .proto and .json files.
Consolidates imports and converts between file formats.
-t, --target Specifies the target format. Also accepts a path to require a custom target.
json JSON representation
json-module JSON representation as a module
proto2 Protocol Buffers, Version 2
proto3 Protocol Buffers, Version 3
static Static code without reflection
static-module Static code without reflection as a module
-p, --path Adds a directory to the include path.
-o, --out Saves to a file instead of writing to stdout.
Module targets only:
-w, --wrap Specifies the wrapper to use. Also accepts a path to require a custom wrapper.
default Default wrapper supporting both CommonJS and AMD
commonjs CommonJS wrapper
amd AMD wrapper
es6 ES6 wrapper
-r, --root Specifies an alternative protobuf.roots name.
-l, --lint Linter configuration. Defaults to protobuf.js-compatible rules:
eslint-disable block-scoped-var, no-redeclare, no-control-regex
Proto sources only:
--keep-case Keeps field casing instead of converting to camel case.
Static targets only:
--no-create Does not generate create functions used for reflection compatibility.
--no-encode Does not generate encode functions.
--no-decode Does not generate decode functions.
--no-verify Does not generate verify functions.
--no-convert Does not generate convert functions like from/toObject
--no-delimited Does not generate delimited encode/decode functions.
--no-beautify Does not beautify generated code.
--no-comments Does not output any JSDoc comments.
usage: pbjs [options] file1.proto file2.json ... (or) other | pbjs [options] -
For production environments it is recommended to bundle all your .proto files to a single .json file, which reduces the number of network requests and parser invocations required:
$> pbjs -t json file1.proto file2.proto > bundle.json
Now, either include this file in your final bundle:
var root = protobuf.Root.fromJSON(require("./bundle.json"));
or load it the usual way:
protobuf.load("bundle.json", function(err, root) {
...
});
ProTip! Documenting your .proto files with /** ... */
-blocks or (trailing) /// ...
lines translates to generated static code.
Likewise, the pbts
command line utility can be used to generate TypeScript definitions from pbjs
-generated static modules.
Generates TypeScript definitions from annotated JavaScript files.
-n, --name Wraps everything in a module of the specified name.
-o, --out Saves to a file instead of writing to stdout.
-m, --main Whether building the main library without any imports.
-g, --global Name of the global object in browser environments, if any.
--no-comments Does not output any JSDoc comments.
usage: pbts [options] file1.js file2.js ... (or) other | pbts [options] -
Both utilities can be used programmatically by providing command line arguments and a callback to their respective main
functions:
var pbjs = require("protobufjs/cli/pbjs");
pbjs.main([ "--target", "json-module", "path/to/myproto.proto" ], function(err, output) {
if (err)
throw err;
// do something with output
});
While .proto and JSON files require the full library (about 18.5kb gzipped), pretty much all code but the relatively short descriptors is shared and all features including reflection and the parser are available.
Static code, on the other hand, requires just the minimal runtime (about 5.5kb gzipped), but generates additional, albeit editable, source code without any reflection features.
There is no difference performance-wise as the code generated statically is pretty much the same as generated at runtime.
Additionally, JSON modules can be used with TypeScript definitions generated for their static counterparts as long as the following conditions are met:
SomeMessage.create(...)
instead of new SomeMessage(...)
(reflection does not provide such a constructor).resolveAll()
must be called once on the root instance to populate these additional properties (JSON modules do this automatically).The package includes a benchmark that tries to compare performance to native JSON as far as this is possible. On an i7-2600K running node 6.9.1 it yields:
benchmarking encoding performance ...
Type.encode to buffer x 547,361 ops/sec ±0.27% (94 runs sampled)
JSON.stringify to string x 310,848 ops/sec ±0.73% (92 runs sampled)
JSON.stringify to buffer x 173,608 ops/sec ±1.51% (86 runs sampled)
Type.encode to buffer was fastest
JSON.stringify to string was 43.5% slower
JSON.stringify to buffer was 68.7% slower
benchmarking decoding performance ...
Type.decode from buffer x 1,294,378 ops/sec ±0.86% (90 runs sampled)
JSON.parse from string x 291,944 ops/sec ±0.72% (92 runs sampled)
JSON.parse from buffer x 256,325 ops/sec ±1.50% (90 runs sampled)
Type.decode from buffer was fastest
JSON.parse from string was 77.4% slower
JSON.parse from buffer was 80.3% slower
benchmarking combined performance ...
Type to/from buffer x 254,126 ops/sec ±1.13% (91 runs sampled)
JSON to/from string x 122,896 ops/sec ±1.29% (90 runs sampled)
JSON to/from buffer x 88,005 ops/sec ±0.87% (89 runs sampled)
Type to/from buffer was fastest
JSON to/from string was 51.7% slower
JSON to/from buffer was 65.3% slower
benchmarking verifying performance ...
Type.verify x 6,246,765 ops/sec ±2.00% (87 runs sampled)
benchmarking message from object performance ...
Type.fromObject x 2,892,973 ops/sec ±0.70% (92 runs sampled)
benchmarking message to object performance ...
Type.toObject x 3,601,738 ops/sec ±0.72% (93 runs sampled)
Note that JSON is a native binding nowadays and as such is about as fast as it possibly can get. So, how can protobuf.js be faster?
verify
on messages to avoid unnecessary overhead where messages are already known to be valid. It's up to the user to call verify
where necessary.You can also run the benchmark ...
$> npm run bench
and the profiler yourself (the latter requires a recent version of node):
$> npm run prof <encode|decode|encode-browser|decode-browser> [iterations=10000000]
Note that as of this writing, the benchmark suite performs significantly slower on node 7.2.0 compared to 6.9.1 because moths.
google/protobuf/descriptor.proto
, options are parsed and presented literally.Long
instance instead of a possibly unsafe JavaScript number (see).To build the library or its components yourself, clone it from GitHub and install the development dependencies:
$> git clone https://github.com/dcodeIO/protobuf.js.git
$> cd protobuf.js
$> npm install
Building the development and production versions with their respective source maps to dist/
:
$> npm run build
Building the documentation to docs/
:
$> npm run docs
Building the TypeScript definition to index.d.ts
:
$> npm run types
By default, protobuf.js integrates into your browserify build-process without requiring any optional modules. Hence:
long
module somewhere in your project. It will be excluded otherwise.License: BSD 3-Clause License
FAQs
Protocol Buffers for JavaScript (& TypeScript).
We found that protobufjs demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 2 open source maintainers collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
PyPI now allows maintainers to archive projects, improving security and helping users make informed decisions about their dependencies.
Research
Security News
Malicious npm package postcss-optimizer delivers BeaverTail malware, targeting developer systems; similarities to past campaigns suggest a North Korean connection.
Security News
CISA's KEV data is now on GitHub, offering easier access, API integration, commit history tracking, and automated updates for security teams and researchers.