pystream-protobuf

pyStream

Python implementation of stream library.

Introduction

This library enables stream processing of protobuf messages (or any serializable objects since v1.6.3); i.e. multiple protobuf messages can be written/read into/from a single stream or file.

It was originally developed to parse/write vg file formats (.vg, .gam, etc). However, it can be used for any arbitrary protocol buffer messages.

Refer to the C++ stream library for more details.

---

NOTE

@vg users: The new version of stream library, now as a part of libvgio, writes a header tag at the start of the stream depending on the output format. For example, headers like b'GAM' or b'VG' can be found before the actual protobuf messages in GAM and VG files repectively. In this case, you should provide the expected value using header keyword argument; e.g. stream.parse('file.gam', vg_pb2.Alignment, header=b'GAM', persistent_header=True) for GAM files (since version v1.6.2).

---

Encoding

The encoding is simple. Messages are written in groups of different sizes. Each group starts with its size; i.e. the number of messages in that group. Then, the size of each message is followed by the encoded message itself. Quoted from Google Protobuf Developer Guide:

The Protocol Buffer wire format is not self-delimiting, so protocol buffer parsers cannot determine where a message ends on their own. The easiest way to solve this problem is to write the size of each message before you write the message itself. When you read the messages back in, you read the size, then read the bytes into a separate buffer, then parse from that buffer.

By default, the stream is considered compressed by GZip. However, uncompressed stream processing is possible by passing gzip=False to any API calls.

Installation

You can install pyStream using pip:

pip install pystream-protobuf

Usage

See Wiki for usage documentation.

Development

In case, you work with the source code and need to build the package:

python setup.py build

The proto files in the test module required to be compiled before running test cases. To do so, it is required to have Google protobuf compiler (>=3.0.2) installed. After installing protobuf compiler, run:

make init

to compile proto files required for test module and then:

make test

to run tests.