protozero

protozero

Minimalistic protocol buffer decoder and encoder in C++.

Designed for high performance. Suitable for writing zero copy parsers and encoders with minimal need for run-time allocation of memory.

Low-level: this is designed to be a building block for writing a very customized decoder for a stable protobuf schema. If your protobuf schema is changing frequently or lazy decoding is not critical for your application then this approach offers no value: just use the C++ API that can be generated with the Google Protobufs protoc program.

Depends

• C++11 compiler
• Tests depend on the Google Protobuf library, but use of Protozero doesn't need it

How it works

The protozero code does not read .proto files used by the usual Protobuf implementations. The developer using protozero has to manually "translate" the .proto description into code. This means there is no way to access any of the information from the .proto description. This results in a few restrictions:

• The names of the fields are not available.
• Enum names are not available, you'll have to use the values they are defined with.
• Default values are not available.
• Field types have to be hardcoded. The library does not know which types to expect, so the user of the library has to supply the right types. Some checks are made using assert(), but mostly the user has to take care of that.

The library will make sure not to overrun the buffer it was given, but basically all other checks have to be made in user code!

Documentation

You have to have a working knowledge of how protocol buffer encoding works.

• Read the tutorial for an introduction on how to use Protozero.
• There is a table of all types and functions in the cheat sheet.
• Macros defined or used by Protozero.
• Read the upgrading instructions if you are upgrading from an older version of Protozero.

Call make doc to build the Doxygen-based reference documentation. (You'll need Doxygen installed.) Then open doc/html/index.html in your browser to read it.

Installation

Call make install to install include files in /usr/include/protozero. Call make install DESTDIR=/usr/local or similar to change install directory.

Limitations

• A protobuf message has to fit into memory completely, otherwise it can not be parsed with this library. There is no streaming support.
• The length of a string, bytes, or submessage can't be more than 2^31-1.
• The Google Protobuf spec documents that a non-repeated field can actually appear several times in a message and the implementation is required to return the value of the last version of that field in this case. pbf_reader.hpp does not enforce this. If this feature is needed in your case, you have to do this yourself.
• There is no specific support for maps but they can be used as described in the "Backwards compatibility" section of https://developers.google.com/protocol-buffers/docs/proto3#maps.

Endianness

Protozero uses a very simplistic test to check the byte order of the system it compiles on. If this check is wrong, you'll get test failures. If this is the case, please open an issue and tell us about your system.

Tests

Extensive tests are included. Call

make test

to build all tests and run them.

See test/README.md for more details about the test.

You can also use gyp to build the reader tests:

gyp gyp/protozero.gyp --depth=. --build=Release
./out/Release/tests

This will clobber the Makefile from the repository! Instead of Release you can use Debug for a debug build.

Coverage report

To get a coverage report compile and link with --coverage:

CXXFLAGS="--coverage" LDFLAGS="--coverage" make test

If you are using g++ use gcov to generate a report (results are in *.gcov files):

gcov -lp test/*tests.o test/t/*/*test_cases.o

If you are using clang++ use llvm-cov instead:

llvm-cov gcov -lp test/*tests.o test/t/*/*test_cases.o

If you are using g++ you can use gcovr to generate nice HTML output:

mkdir -p coverage
gcovr -r . --html --html-details -o coverage/index.html

Open coverage/index.html in your browser to see the report.

Cppcheck

For extra checks with Cppcheck you can call

make check

Who is using Protozero?

• Carmen
• Libosmium
• Mapbox GL Native
• Mapbox Vector Tile library
• Mapnik
• OSRM
• Tippecanoe

Are you using Protozero? Tell us! Send a pull request with changes to this README.

Changelog

[1.4.5] - 2016-11-18

Fixed

• Undefined behaviour in packed fixed iterator. As a result, the macro PROTOZERO_DO_NOT_USE_BARE_POINTER is not used any more.