protoscript

ProtoScript

A Protobuf runtime and code generation tool for JavaScript and TypeScript

What is this? 🧐

ProtoScript is a protocol buffers runtime and code generation tool for JavaScript, written in TypeScript.

ProtoScript consists of two parts:

• Runtime. This is nearly identical to google-protobuf, but significantly smaller (9.6KB vs 46KB gzipped) and written in ESM to enable dead-code elimination.

• Code generation. This is a replacement of protoc's JavaScript generation that generates idiomatic JavaScript code, JSON serializers/deserializers, and includes TSDoc comments.

If you're looking for an RPC framework, you may be interested in TwirpScript.

Highlights 🛠

• Idiomatic JavaScript / TypeScript code. None of the Java idioms that protoc --js_out generates such as the List suffix naming for repeated fields, Map suffix for maps, or the various getter and setter methods. ProtoScript generates and consumes plain JavaScript objects over classes. Compare the TypeScript example to the protoc example.

• In-editor API documentation. Comments in your .proto files become TSDoc comments in the generated code and will show inline documentation in supported editors.

• JSON Serialization/Deserialization. Unlike protoc, ProtoScript's code generation generates JSON serialization and deserialization methods.

• Small. ProtoScript's runtime and generated code are built with tree shaking to minimize bundle sizes. This results in a significantly smaller bundle size than google-protobuf. ProtoScript's runtime is 67KB (9.6KB gzipped) compared to google-protobuf's 231KB (46KB gzipped).

• Isomorphic. ProtoScript's generated serializers/deserializers can be consumed in the browser or Node.js runtimes.

• No additional runtime dependencies.

Installation 📦

• Install the protocol buffers compiler:

  MacOS: brew install protobuf

  Linux: apt install -y protobuf-compiler

  Windows: choco install protoc

  Or install from a precompiled binary.

• Add this package to your project: npm install protoscript or yarn add protoscript

Requirements ⚠️

• Node.js v16 or greater
• TypeScript v4.7 or greater when using TypeScript
• webpack setup when using a compiler other than TypeScript (like webpack)

Examples 🚀

npx

npx protoscript

cli

protoc \
  --plugin=protoc-gen-protoscript=./node_modules/protoscript/compiler.js
  --protoscript_out=. \
  --protoscript_opt=language=typescript \

Note: Windows users replace ./node_modules/protoscript/compiler.js above with ./node_modules/protoscript/compiler.cmd

Buf

ProtoScript can be used with Buf.

buf.gen.yaml

version: v1
plugins:
  - name: protoc-gen-protoscript
    path: ./node_modules/protoscript/compiler.js
    out: .
    opt:
      - language=typescript
    strategy: all

Working with other tools

TypeScript

As a design goal, ProtoScript should always work with the strictest TypeScript compiler settings. If your generated ProtoScript files are failing type checking, please open an issue.

Linters

ProtoScript does not make any guarantees for tools like linters and formatters such as prettier or eslint. It generally does not make sense to run these tools against code generation artifacts, like the .pb.ts or .pb.js files generated by ProtoScript. This is because:

• The permutations of preferences and plugins for these tools quickly explode beyond what is feasible for a single tool to target. There are always new tools that could be added as well.
• The code is generated automatically, and not all rules are auto fixable. This means there are cases that would always require manual intervention by the user.
• Autogenerated code is readonly, and expected to be correct. Autogenerated code has a much difference maintenance cycle than code written by hand, and should generally be treated as a binary or a dependency. You don't lint your node_modules!

Configuration 🛠

ProtoScript aims to be zero config, but can be configured via the cli interface, or when using the npx protoscript command, by creating a proto.config.mjs (or .js or .cjs) file in your project root.

Name	Description	Type
root	The root directory. `.proto` files will be searched under this directory, and `proto` import paths will be resolved relative to this directory. ProtoScript will recursively search all subdirectories for `.proto` files. Defaults to the project root. Example: If we have the following project structure: /src A.proto B.proto Default: A.proto would import B.proto as follows: import "src/B.proto"; Setting root to src: // proto.config.mjs /** @type {import('protoscript').Config} */ export default { root: "src", }; A.proto would import B.proto as follows: import "B.proto"; TypeScript projects will generally want to set this value to match their rootDir, particularly when using Protocol Buffers Well-Known Types so that the generated well-known type files are under the rootDir.	string (filepath)
exclude	An array of patterns that should be skipped when searching for `.proto` files. Example: If we have the following project structure: /src /foo A.proto /bar B.proto Setting exclude to ["/bar/"]: // proto.config.mjs /** @type {import('protoscript').Config} */ export default { exclude: ["/bar/"] } Will only process A.proto (B.proto) will be excluded from ProtoScript's code generation.	string[] (RegExp pattern)
dest	The destination folder for generated files. Defaults to colocating generated files with the corresponding proto definition. If we have the following project structure: /src A.proto B.proto ProtoScript will generate the following: /src A.proto A.pb.ts B.proto B.pb.ts Setting dest to out will generate the following: // proto.config.mjs /** @type {import('protoscript').Config} */ export default { dest: "out", } /src A.proto B.proto /out /src A.pb.ts B.pb.ts Note that the generated directory structure will mirror the proto paths exactly as is, only nested under the dest directory. If you want to change this, for instance, to omit src from the out directory above, you can set the root. Setting root to src (in addition to setting dest to out) will generate the following: // proto.config.mjs /** @type {import('protoscript').Config} */ export default { root: "src", dest: "out", } /src A.proto B.proto /out A.pb.ts B.pb.ts	string (filepath)
language	Whether to generate JavaScript or TypeScript. If omitted, ProtoScript will attempt to autodetect the language by looking for a tsconfig.json in the project root. If found, ProtoScript will generate TypeScript, otherwise JavaScript.	javascript | typescript
json	JSON serializer options. emitFieldsWithDefaultValues - Fields with default values are omitted by default in proto3 JSON. Setting this to true will serialize fields with their default values. useProtoFieldName - Field names are converted to lowerCamelCase by default in proto3 JSON. Setting this to true will use the proto field name as the JSON key when serializing JSON. Either way, Proto3 JSON parsers are required to accept both the converted lowerCamelCase name and the proto field name. See https://developers.google.com/protocol-buffers/docs/proto3#json for more context.	{ emitFieldsWithDefaultValues?: boolean, useProtoFieldName?: boolean }
typecript	TypeScript options. emitDeclarationOnly - Only emit TypeScript type definitions.	{ emitDeclarationOnly?: boolean }

JSON

ProtoScript's JSON serialization/deserialization implements the proto3 specification. This is nearly complete, but still in progress.

ProtoScript will serialize JSON keys as lowerCamelCase versions of the proto field. Per the proto3 spec, the runtime will accept both lowerCamelCase and the original proto field name when deserializing. You can provide the json_name field option to specify an alternate key name. When doing so, the runtime will encode JSON messages using the the json_name as the key, and will decode JSON messages using the json_name if present, otherwise falling back to the lowerCamelCase name and finally to the original proto field name.

Webpack Setup

If you're using a compiler other than TypeScript, such as webpack, this outstanding issue in TypeScript adds an additional hurdle.

The short summary of the issue:

• Extensions are required for ESM imports.
• Extension-less imports have sidestepped this TypeScript issue, but extensions are required for ESM adoption.
• The TypeScript compiler requires .js extensions for imports.
• Compilers other than TypeScript expect .ts extensions for imports.

The TypeScript team is investigating improving this issue in their 4.9 plan

Webpack users can use extensionAlias to solve this problem:

module.exports = {
+  resolve: {
+    extensionAlias: {
+      ".js": [".ts", ".js"],
+    },
+  },
};

For more context, see TypeScript#37582 and Webpack#13252.

Contributing 👫

PR's and issues welcomed! For more guidance check out CONTRIBUTING.md

Licensing 📃

See the project's MIT License.

Changelog

v0.0.10

• Change configuration file format. Now, the configuration file is JS instead of JSON. This provides better discoverability and type checking for TypeScript users.

The following .protoscript.json:

{
  "root": "src",
};

Would be renamed to proto.config.mjs and changed to the following:

/** @type {import('protoscript').Config} */
export default {
  root: "src",
};

• Use relative file path for determining path to compiler instead of hard coding from project root. This should interop better with more exotic package tooling and repo setup.

• Fix: Improved map detection. Previously field types suffixed with Entry were incorrectly flagged as maps. This has been fixed.