rawproto

rawproto

Guess structure of protobuf binary from raw data, query binary protobuf without the schema, and output guessed JSON or schema, some CLI utils, and a web tool for exploring raw protobuf.

You can explore your proto binary data here. Use it to view, generate proto/json files, or select how to parse fields.

If you are coming form an older version, or anothe rlibrary, check out migration instructions.

installation

npm i rawproto will add this to your project.

You can also use npx rawproto to run the CLI.

If you just want the CLI, and don't use node, you can also find standalone builds here.

usage

CLI

Install it in your path with npm i -g rawproto or use it 1-off with npx rawproto. Get help with rawproto --help

code

You can use it in code like this:

import { readFile } from 'fs/promises'
import RawProto from 'rawproto'

// load proto
const proto = new RawProto(await readFile('data.pb'))

// get a single field, without parsing the whole tree
console.log(proto.query('1.2.4.10.5:string'))

// you can also pull things like they are arrays/values
console.log(proto['1'][0]['2'][0]['4'][0]['10'].map(r => r['5'][0].string ))

// guess to decode as JS object
console.log(proto.toJS())

// guess to generate .proto file string
console.log(proto.toProto())

// walk over all fields recursively, calling your callback.
const mydata = proto.walk((field) => {
  console.log(field)

  // just do whatever it normally does to make JS-object
  return walkerJS(field)
})

types

Protobuf encodes several different possible types for every wire-type. In this lib, we guess the type based on some context-clues, but it will never be perfect, without hand-tuning. Here are the possible types we support:

VARINT - int, bool, string
FIXED64 - uint, int, bytes, float, string
LEN - string, bytes, packedIntVar, packedInt32, packedInt64, string
FIXED32 - int, uint, bytes, float, string

• You can use any protobuf scalar type-name.
• You can use raw for any type to get the raw field with bytes + meta.
• Groups are treated as repeated LEN message-fields
• LEN will try to be parsed as sub-tree, but you can override with other types in query (for example if it tries to make a sub-message with part of a string)

query-map

Many things (ui, toJS, toProto, cli) use queryMap which is just a map of name to path:type. Here is one that works well with hearthstone test data:

{
  "id": "1.2.4.1:string",
  "title": "1.2.4.5:string",
  "company": "1.2.4.6:string",
  "description": "1.2.4.7:string",

  "media": "1.2.4.10",
  
  "dimensions": "1.2.4.10.2",
  "width": "1.2.4.10.2.3:uint",
  "height": "1.2.4.10.2.4:uint",

  "url": "1.2.4.10.5:string",
  "type": "1.2.4.10.1:uint",
  "bg": "1.2.4.10.15:string"
}

You can use any types, from above, and set the name to whatever you want.

migration

I used to have the functionality of this lib split up into several other projects. Here is migration instructions, if you want to update to this one (recommended):

• protobuf-decoder - just use site. The code is here
• rawprotoparse - this originally would create JSON from protobuf binary. If you were using this as-is, it had a lot of options, which have been merged into either toJS (see tests for examples.) It may require a little bit more custom-code, if you were not using it with defaults, but overall should work better, and merges shared code that was in both libs. Main thing is that regular toJS, without a custom-mapper, will make all values an array, since it's possible for any field ID to be found multiple times.
• newrawprotoparser - this was some of the start of ideas for this. No one is probly using this. Essentially, it's the same stuff in path
• protoquery - this was some of the start of ideas for this. No one is probly using this. Essentially it's the same stuff in query
• rawproto - This lib used to be able to do JSON and generate proto, and provided a different CLI. You should be able to use the new APIs to accomplish all the same stuff, but it may require a bit of a change to your code. Have a look at the unit-tests, to get an idea of how it works.