@bufbuild/buf

What is @bufbuild/buf?

@bufbuild/buf is an npm package that provides tools for working with Protocol Buffers (protobufs). It helps in generating code, linting, and managing protobuf files efficiently. The package is part of the Buf ecosystem, which aims to improve the developer experience when working with protobufs.

What are @bufbuild/buf's main functionalities?

Code Generation

This feature allows you to generate code from your protobuf files. You can specify the input directory containing your .proto files, the output directory for the generated code, and the plugins for the desired languages or frameworks.

const { generate } = require('@bufbuild/buf');

async function generateCode() {
  await generate({
    input: 'path/to/proto/files',
    output: 'path/to/output/directory',
    plugins: ['typescript', 'grpc-web']
  });
}

generateCode();

Linting

Linting helps you ensure that your protobuf files adhere to best practices and style guidelines. This feature scans your .proto files and reports any issues or warnings.

const { lint } = require('@bufbuild/buf');

async function lintProtos() {
  const result = await lint({
    input: 'path/to/proto/files'
  });
  console.log(result);
}

lintProtos();

Breaking Change Detection

This feature allows you to detect breaking changes between different versions of your protobuf files. It compares the current version of your .proto files against a previous version and reports any breaking changes.

const { breaking } = require('@bufbuild/buf');

async function checkBreakingChanges() {
  const result = await breaking({
    against: 'path/to/previous/proto/files',
    input: 'path/to/current/proto/files'
  });
  console.log(result);
}

checkBreakingChanges();

Other packages similar to @bufbuild/buf

protobufjs

protobufjs is a popular library for working with Protocol Buffers in JavaScript. It provides a comprehensive set of tools for parsing, serializing, and generating protobuf messages. Unlike @bufbuild/buf, protobufjs focuses more on runtime operations and less on linting or breaking change detection.

grpc-tools

grpc-tools is a package that provides tools for working with gRPC and Protocol Buffers. It includes a code generator for various languages and a set of utilities for managing gRPC services. While it offers code generation similar to @bufbuild/buf, it does not provide linting or breaking change detection features.

protoc-gen-ts

protoc-gen-ts is a plugin for the Protocol Buffers compiler (protoc) that generates TypeScript code from .proto files. It is specifically focused on TypeScript code generation and does not offer the broader set of features like linting or breaking change detection that @bufbuild/buf provides.

README

Buf

The buf CLI is the best tool for working with Protocol Buffers. It provides:

• A linter that enforces good API design choices and structure.
• A breaking change detector that enforces compatibility at the source code or wire level.
• A generator that invokes your plugins based on configurable templates.
• A formatter that formats your Protobuf files in accordance with industry standards.
• Integration with the Buf Schema Registry, including full dependency management.

Installation

Homebrew

You can install buf using Homebrew (macOS or Linux):

brew install bufbuild/buf/buf

This installs:

• The buf, protoc-gen-buf-breaking, and protoc-gen-buf-lint binaries
• Shell completion scripts for Bash, Fish, Powershell, and zsh

Other methods

For other installation methods, see our official documentation, which covers:

• Installing buf via npm
• Installing buf on Windows
• Using buf as a Docker image
• Installing as a binary, from a tarball, and from source through GitHub Releases
• Verifying releases using a minisign public key

Usage

Buf's help interface provides summaries for commands and flags:

buf --help

For more comprehensive usage information, consult Buf's documentation, especially these guides:

• buf breaking
• buf build
• buf generate
• buf lint
• buf format
• buf registry (for using the BSR)

CLI breaking change policy

We will never make breaking changes within a given major version of the CLI. Once buf reaches v1.0, you can expect no breaking changes until v2.0. But as we have no plans to ever release a v2.0, we will likely never break the buf CLI.

This breaking change policy does not apply to commands behind the buf beta gate, and you should expect breaking changes to commands like buf beta registry. The policy does go into effect, however, when those commands or flags are elevated out of beta.

Our goals for Protobuf

Buf's goal is to replace the current paradigm of API development, centered around REST/JSON, with a schema-driven paradigm. Defining APIs using an IDL provides numerous benefits over REST/JSON, and Protobuf is by far the most stable and widely adopted IDL in the industry. We've chosen to build on this widely trusted foundation rather than creating a new IDL from scratch.

But despite its technical merits, actually using Protobuf has long been more challenging than it needs to be. The Buf CLI and the BSR are the cornerstones of our effort to change that for good and to make Protobuf reliable and easy to use for service owners and clients alike—in other words, to create a modern Protobuf ecosystem.

While we intend to incrementally improve on the buf CLI and the BSR, we're confident that the basic groundwork for such an ecosystem is already in place.

The Buf Schema Registry

The Buf Schema Registry (BSR) is a SaaS platform for managing your Protobuf APIs. It provides a centralized registry and a single source of truth for all of your Protobuf assets, including not just your .proto files but also remote plugins. Although the BSR provides an intuitive browser UI, buf enables you to perform most BSR-related tasks from the command line, such as pushing Protobuf sources to the registry and managing users and repositories.

The BSR is not required to use buf. We've made the core features of the buf CLI available to all Protobuf users.

More advanced CLI features

While buf's core features should cover most use cases, we've included some more advanced features to cover edge cases:

• Automatic file discovery. Buf walks your file tree and builds your .proto files in accordance with your supplied build configuration, which means that you no longer need to manually specify --proto_paths. You can still, however, specify .proto files manually through CLI flags in cases where file discovery needs to be disabled.
• Fine-grained rule configuration for linting and breaking changes. While we do have recommended defaults, you can always select the exact set of rules that your use case requires, with 40 lint rules and 53 breaking change rules available.
• Configurable error formats for CLI output. buf outputs information in file:line:column:message form by default for each lint error and breaking change it encounters, but you can also select JSON and, in the near future, JUnit output.
• Editor integration driven by buf's granular error output. We currently provide linting integrations for both Vim and Visual Studio Code but we plan to support other editors, such as Emacs and JetBrains IDEs like IntelliJ and GoLand, in the future.
• Universal Input targeting. Buf enables you to perform actions like linting and breaking change detection not just against local .proto files but also against a broad range of other Inputs, such as tarballs and ZIP files, remote Git repositories, and pre-built image files.
• Speed. Buf's internal Protobuf compiler compiles your Protobuf sources using all available cores without compromising deterministic output, which is considerably faster than protoc. This allows for near-instantaneous feedback, which is of special importance for features like editor integration.

Next steps

Once you've installed buf, we recommend completing the Tour of Buf, which provides a broad but hands-on overview of the core functionality of both the CLI and the BSR. The tour takes about 10 minutes to complete.

After completing the tour, check out the remainder of the docs for your specific areas of interest.

Community

For help and discussion around Protobuf, best practices, and more, join us on Slack.

For updates on the Buf CLI, follow this repo on GitHub.

For feature requests, bugs, or technical questions, email us at [e-mail]. For general inquiries or inclusion in our upcoming feature betas, email us at [e-mail].

Changelog

[v1.32.0] - 2024-05-16

• Add version v2 for buf.yaml and buf.gen.yaml configuration files.
• Add buf config migrate to migrate configuration files to the latest version (now v2).
• Move buf mod init to buf config init. buf mod init is now deprecated.
• Move buf mod ls-lint-rules to buf config ls-lint-rules. buf mod ls-lint-rules is now deprecated.
• Move buf mod ls-breaking-rules to buf config ls-breaking-rules. buf mod ls-breaking-rules is now deprecated.
• Move buf mod prune to buf dep prune. buf mod prune is now deprecated.
• Move buf mod update to buf dep update. buf mod update is now deprecated.
• Move buf mod {clear-cache,cc} to buf registry cc. buf mod {clear-cache,cc} is now deprecated.
• Move buf beta graph to stable as buf dep graph.
• Change the default visibility of buf push --create-visibility to private when the --create flag is set. Users are no longer required to set --create-visibility when running buf push --create.
• Add buf push --label, which allows users to set labels when pushing new commits to the BSR.
• Add buf push --source-control-url, which allows users to associate commits pushed to the BSR with a URL to a source code repository.
• Add buf push --create-default-label, which allows users to set a default label for a repository when calling buf push --create.
• Add buf push --git-metadata, which automatically sets appropriate --label, --source-control-url, and --create-default-label flags based on the current Git repository.
• Add buf convert --validate to apply protovalidate rules to incoming messages specified with --from.
• Deprecate buf mod open.
• Delete buf beta migrate-v1beta1 This is now replaced with buf config migrate.
• Add buf registry sdk version to get the version of a Generated SDK for a module and plugin.
• Add buf beta registry archive and buf beta registry unarchive commands for archiving and unarchiving labels on the BSR.
• Add support for Protobuf Editions. This allows buf to be used with sources that use edition 2023, instead of proto2 or proto3 syntax. This also updates the protoc-gen-buf-breaking and protoc-gen-buf-lint Protobuf plugins to support files that use edition 2023.
• Update buf breaking rules to work with Protobuf Editions. To support Editions, some rules have been deprecated and replaced with Editions-aware rules. All deprecated rules continue to work for existing users.
  • FIELD_SAME_CTYPE has been replaced with FIELD_SAME_CPP_STRING_TYPE, which considers both ctype field options and new (pb.cpp).string_type features when deciding on backwards compatibility.
  • FIELD_SAME_LABEL has been replaced with three rules that all check "cardinality". The new rules can distinguish between maps and other repeated fields and between implicit and explicit field presence. The new rules are:
    1. FIELD_SAME_CARDINALITY in the FILE and PACKAGE categories.
    2. FIELD_WIRE_COMPATIBLE_CARDINALITY in the WIRE category.
    3. FIELD_WIRE_JSON_COMPATIBLE_CARDINALITY in the WIRE_JSON category.
  • FILE_SAME_JAVA_STRING_CHECK_UTF8 has been replaced with FIELD_SAME_JAVA_UTF8_VALIDATION, which considers both the java_string_check_utf8 file option and (pb.java).utf8_validation features when deciding on backwards compatibility.
  • Add to the existing FILE_SAME_SYNTAX rule with a few related rules that can catch the same sort of compatibility issues, but in an Editions source file that changes feature values:
    1. MESSAGE_SAME_JSON_FORMAT and ENUM_SAME_JSON_FORMAT catch changes to the json_format feature, which controls whether support for the JSON format is best-effort or properly supported. When supported, the compiler performs more checks relating to field name collisions for the JSON format as well as for FieldMask usage.
    2. FIELD_SAME_UTF8_VALIDATION catches changes to the utf8_validation feature, which controls validation of string values.
    3. ENUM_SAME_TYPE catches changes to an enum's type, open vs. closed.
• Add support for extensions to buf breaking. All existing rules for fields are now applied to extensions, except for FIELD_NO_DELETE (and its variants). There are also new EXTENSION_NO_DELETE and PACKAGE_EXTENSION_NO_DELETE rules for catching deletions of an extension. The new rules are not active by default in existing v1 and v1beta1 configurations, for backwards-compatibility reasons. Migrate your config to v2 to use them.
• Add support for top-level extensions to buf lint. It previously only checked extensions that were defined inside of messages.
• Add a new FIELD_NOT_REQUIRED lint rule that prevents use of required in proto2 files and of features.field_presence = LEGACY_REQUIRED in Editions files. This new rule is not active by default in existing v1 and v1beta1 configurations, for backwards-compatibility reasons. Migrate your config to v2 to use them.