protoscript

ProtoScript

A Protobuf runtime and code generation tool for JavaScript and TypeScript

What is this? 🧐

ProtoScript is a protocol buffers runtime and code generation tool for JavaScript and TypeScript.

ProtoScript consists of two parts:

1. A runtime. The interface is nearly identical to google-protobuf, but under the hood it is significantly smaller (9.6KB vs 46KB gzipped) and written in ESM to enable dead-code elimination. If you only use the generated JSON serializers, the protobuf runtime will be eliminated from the final build, resulting in an even smaller runtime (less than 1 KB).

2. A code generator. This is a replacement of protoc's JavaScript generation that generates idiomatic JavaScript code, JSON serializers/deserializers, and includes TSDoc comments.

If you're looking for an RPC framework, you may be interested in TwirpScript, which uses ProtoScript under the hood to implement Twirp.

Use cases

• Network communication: Protobuf can be used to define APIs and data contracts in a language-independent manner. This makes it easier to share and maintain APIs among different services and platforms. Services written in various programming languages can be autogenerated and exchange data in a language-agnostic manner, using protobuf or JSON as the wire format.
• Data storage: store and retrieve structured data in a compact binary format. protobuf's binary format is more space-efficient than JSON, XML and other formats.

Highlights 🛠

1. Idiomatic JavaScript and TypeScript code. ProtoScript generates and consumes plain JavaScript objects over classes. It does not generate any of Java idioms from protoc --js_out such as the List suffix naming for repeated fields, Map suffix for maps, or the various getter and setter methods. Compare the generated code from ProtoScript's TypeScript example to the protoc example.

2. In-editor API documentation. Comments in your .proto files become TSDoc comments in the generated code and will show inline documentation in supported editors.

3. JSON Serialization/Deserialization. Unlike protoc, ProtoScript's code generation also generates JSON serialization and deserialization methods.

4. Small. ProtoScript's runtime and generated code are built to support tree shaking to minimize bundle sizes. This results in a significantly smaller bundle size than google-protobuf. ProtoScript's runtime is 67KB (9.6KB gzipped) compared to google-protobuf's 231KB (46KB gzipped). If you only use the generated JSON serializers, the protobuf runtime will be eliminated from the final build, resulting in a substantially smaller runtime (less than 1 KB).

5. Isomorphic. ProtoScript's generated serializers/deserializers can be consumed in the browser or Node.js runtimes.

6. No additional runtime dependencies.

Example

If you have the following protobuf defined in user.proto:

syntax = "proto3";

message User {
  string first_name = 1;
  string last_name = 2;
  bool active = 3;
  User manager = 4;
  repeated string locations = 5;
  map<string, string> projects = 6;
}

Using the User generated by ProtoScript will look like this:

import { User, UserJSON } from "./user.pb.js";

const user = {
  firstName: "Harry",
  lastName: "Potter",
  active: true,
  locations: ["Hogwarts"],
  projects: { DA: "Dumbledore's Army },
  manager: {
    firstName: "Albus",
    lastName: "Dumbledore",
  }
}

// protocol buffers
const bytes = User.encode(user);
console.log(bytes); // Uint8Array

const userFromBytes = User.decode(bytes);
console.log(userFromBytes); // our user above

// json
const json = UserJSON.encode(user)
console.log(json); // json string

const userFromJSON = UserJSON.decode(json);
console.log(userFromJSON); // our user above

// ProtoScript generates and consumes plain JavaScript objects (POJOs) over classes. If you want to generate a full message
// with default fields, you can use the #initialize method on the generated message class:
const user = User.initialize();
console.log(user);

// the generated object is just a POJO. You can encode any POJO as protobuf or json as long as it conforms to a subset of the message interface defined in your .proto:
console.log(User.encode(user))
console.log(UserJSON.encode(user))

Installation 📦

1. Install the protocol buffers compiler:

   MacOS: brew install protobuf

   Linux: apt install -y protobuf-compiler

   Windows: choco install protoc

   Or install from a precompiled binary.

2. Add this package to your project: npm install protoscript or yarn add protoscript

Requirements ⚠️

• Node.js v16 or greater
• TypeScript v4.7 or greater when using TypeScript

Examples 🚀

npx

npx protoscript

cli

protoc \
  --plugin=protoc-gen-protoscript=./node_modules/protoscript/compiler.js
  --protoscript_out=. \
  --protoscript_opt=language=typescript \

Note: Windows users replace ./node_modules/protoscript/compiler.js above with ./node_modules/protoscript/compiler.cmd

Buf

ProtoScript can be used with Buf.

buf.gen.yaml

version: v1
plugins:
  - name: protoc-gen-protoscript
    path: ./node_modules/protoscript/dist/compiler.js
    out: .
    opt:
      - language=typescript
    strategy: all

Working with other tools

TypeScript

As a design goal, ProtoScript should always work with the strictest TypeScript compiler settings. If your generated ProtoScript files are failing type checking, please open an issue.

Linters

ProtoScript does not make any guarantees for tools like linters and formatters such as prettier or eslint. It generally does not make sense to run these tools against code generation artifacts, like the .pb.ts or .pb.js files generated by ProtoScript. This is because:

1. The permutations of preferences and plugins for these tools quickly explode beyond what is feasible for a single tool to target. There are always new tools that could be added as well.
2. The code is generated automatically, and not all rules are auto fixable. This means there are cases that would always require manual intervention by the user.
3. Autogenerated code is readonly, and expected to be correct. Autogenerated code has a much difference maintenance cycle than code written by hand, and should generally be treated as a binary or a dependency. You don't lint your node_modules!

Configuration 🛠

ProtoScript aims to be zero config, but can be configured via the cli interface, or when using the npx protoscript command, by creating a proto.config.mjs (or .js or .cjs) file in your project root.

Name	Description	Type
root	The root directory. `.proto` files will be searched under this directory, and `proto` import paths will be resolved relative to this directory. ProtoScript will recursively search all subdirectories for `.proto` files. Defaults to the project root. Example: If we have the following project structure: /src A.proto B.proto Default: A.proto would import B.proto as follows: import "src/B.proto"; Setting root to src: // proto.config.mjs /** @type {import('protoscript').Config} */ export default { root: "src", }; A.proto would import B.proto as follows: import "B.proto"; TypeScript projects will generally want to set this value to match their rootDir, particularly when using Protocol Buffers Well-Known Types so that the generated well-known type files are under the rootDir.	string (filepath)
exclude	An array of patterns that should be skipped when searching for `.proto` files. Example: If we have the following project structure: /src /foo A.proto /bar B.proto Setting exclude to ["/bar/"]: // proto.config.mjs /** @type {import('protoscript').Config} */ export default { exclude: ["/bar/"] } Will only process A.proto (B.proto) will be excluded from ProtoScript's code generation.	string[] (RegExp pattern)
dest	The destination folder for generated files. Defaults to colocating generated files with the corresponding proto definition. If we have the following project structure: /src A.proto B.proto ProtoScript will generate the following: /src A.proto A.pb.ts B.proto B.pb.ts Setting dest to out will generate the following: // proto.config.mjs /** @type {import('protoscript').Config} */ export default { dest: "out", } /src A.proto B.proto /out /src A.pb.ts B.pb.ts Note that the generated directory structure will mirror the proto paths exactly as is, only nested under the dest directory. If you want to change this, for instance, to omit src from the out directory above, you can set the root. Setting root to src (in addition to setting dest to out) will generate the following: // proto.config.mjs /** @type {import('protoscript').Config} */ export default { root: "src", dest: "out", } /src A.proto B.proto /out A.pb.ts B.pb.ts	string (filepath)
language	Whether to generate JavaScript or TypeScript. If omitted, ProtoScript will attempt to autodetect the language by looking for a tsconfig.json in the project root. If found, ProtoScript will generate TypeScript, otherwise JavaScript.	javascript | typescript
json	JSON serializer options. emitFieldsWithDefaultValues - Fields with default values are omitted by default in proto3 JSON. Setting this to true will serialize fields with their default values. useProtoFieldName - Field names are converted to lowerCamelCase by default in proto3 JSON. Setting this to true will use the proto field name as the JSON key when serializing JSON. Either way, Proto3 JSON parsers are required to accept both the converted lowerCamelCase name and the proto field name. See https://developers.google.com/protocol-buffers/docs/proto3#json for more context.	{ emitFieldsWithDefaultValues?: boolean, useProtoFieldName?: boolean }
typecript	TypeScript options. emitDeclarationOnly - Only emit TypeScript type definitions.	{ emitDeclarationOnly?: boolean }

JSON

ProtoScript's JSON serialization/deserialization implements the proto3 specification. If you find a discrepancy, please open an issue.

ProtoScript will serialize JSON keys as lowerCamelCase versions of the proto field. Per the proto3 spec, the runtime will accept both lowerCamelCase and the original proto field name when deserializing. You can provide the json_name field option to specify an alternate key name. When doing so, the runtime will encode JSON messages using the the json_name as the key, and will decode JSON messages using the json_name if present, otherwise falling back to the lowerCamelCase name and finally to the original proto field name.

Contributing 👫

PR's and issues welcomed! For more guidance check out CONTRIBUTING.md

Licensing 📃

See the project's MIT License.