py-lz4framed

Overview

This is an LZ4-frame compression library for Python v3.2+ (and 2.7+), bound to Yann Collet's LZ4 C implementation.

Installing / packaging

# To get from PyPI
pip3 install py-lz4framed

# To only build extension modules inline (e.g. in repository)
python3 setup.py build_ext -i

# To build & install globally
python3 setup.py install

Notes

• The above as well as all other python3-using commands should also run with v2.7+
• This module is also available via Anaconda (conda-forge) (with binaries for Linux, OSX and Windows)
• PyPI releases are signed with the Iotic Labs Software release signing key

Usage

Single-function operation:

import lz4framed

compressed = lz4framed.compress(b'binary data')

uncompressed = lz4framed.decompress(compressed)

To iteratively compress (to a file or e.g. BytesIO instance):

with open('myFile', 'wb') as f:
    # Context automatically finalises frame on completion, unless an exception occurs
    with Compressor(f) as c:
        try:
            while (...):
               c.update(moreData)
        except Lz4FramedNoDataError:
            pass

To decompress from a file-like object:

with open('myFile', 'rb') as f:
    try:
        for chunk in Decompressor(f):
           decoded.append(chunk)
    except Lz4FramedNoDataError:
        # Compress frame data incomplete - error case
        ...

See also lz4framed/__main__.py for example usage.

Documentation

import lz4framed
print(lz4framed.__version__, lz4framed.LZ4_VERSION, lz4framed.LZ4F_VERSION)
help(lz4framed)

Command-line utility

python3 -mlz4framed
USAGE: lz4framed (compress|decompress) (INFILE|-) [OUTFILE]

(De)compresses an lz4 frame. Input is read from INFILE unless set to '-', in
which case stdin is used. If OUTFILE is not specified, output goes to stdout.

Tests

Static

This library has been checked using flake8 and pylint, using a modified configuration - see pylint.rc and flake8.cfg.

Unit

python3 -m unittest discover -v .

Why?

The only existing lz4-frame interoperable implementation I was aware of at the time of writing (lz4tools) had the following limitations:

• Incomplete implementation in terms of e.g. reference & memory leaks on failure
• Lack of unit tests
• Not thread safe
• Does not release GIL during low level (de)compression operations
• Did not address the requirements for an external project