Huge News!Announcing our $40M Series B led by Abstract Ventures.Learn More →
fast-zlib - npm Package Compare versions
Comparing version 1.0.1 to 1.0.2
@@ -53,3 +53,3 @@ let zlib = require("zlib"); | ||
| if(unsafe) { z._defaultFlushFlag = true; } | ||
| d.zlib = z; | ||
| d._zlib = z; | ||
| return d; | ||
@@ -56,0 +56,0 @@ } |
| { | ||
| "name": "fast-zlib", | ||
| "version": "1.0.1", | ||
| "version": "1.0.2", | ||
| "description": "Synchronous shared context compression with node's native zlib", | ||
@@ -5,0 +5,0 @@ "main": "index.js", |
182
README.md
| # fast-zlib | ||
| This package is a simple wrapper for zlib's `_processChunk()` method in Node.js and exposes an easy to use high level API for synchronous shared context compression. | ||
| This package is a simple `zlib` wrapper for Node.js that exposes functions for synchronous shared context compression. | ||
@@ -11,42 +11,24 @@ Method inspired from [isaacs/minizlib](https://github.com/isaacs/minizlib) | ||
| Node's native zlib module does not offer a public API to perform this task synchronously and instead offers an asynchronous API using transform streams to be as non-blocking as possible. Because zlib itself is synchronous and does not depend on anything other than cpu, artificially making it asynchronous ends up introducing problems with performance, high latency and and memory fragmentation, especially with a high volume of small chunks of data. (see [ws#1369](https://github.com/websockets/ws/issues/1369) and [node#8871](https://github.com/nodejs/node/issues/8871)) | ||
| Node's native zlib module does not offer a public API to perform this task synchronously and instead offers an asynchronous API using transform streams. Because zlib itself is synchronous and does not depend on anything other than cpu, artificially making it asynchronous may cause problems with performance, high latency and memory fragmentation, especially with a high volume of small chunks of data. (see [ws#1369](https://github.com/websockets/ws/issues/1369) and [node#8871](https://github.com/nodejs/node/issues/8871)) | ||
| Node does however include all the necessary tools and functionality in its private and undocumented APIs, which this package makes use of to provide an easy and fast way to synchronously process chunks in a shared zlib context. | ||
| Node does however include all the necessary tools and functionality in its private and undocumented APIs, for instance its `_processChunk()` method, which this package makes use of to provide an easy and fast way to synchronously process chunks in a shared zlib context. | ||
| ## Usage | ||
| ## Docs | ||
| Usage is similar to any other synchronous compression library, compress a chunk, then decompress it elsewhere. | ||
| This package is essentially a high-order function that returns a compressor or a decompressor function. | ||
| ```js | ||
| let zlib = require("fast-zlib"); | ||
| ### fastzlib(method[, options]) | ||
| let deflate = zlib("deflate"); | ||
| let inflate = zlib("inflate"); | ||
| - `method` (String): A string representing the zlib class to create | ||
| - `options` (Object): An options object for the zlib instance | ||
| - Returns (Function): A compressor or a decompressor function | ||
| let data = "123456789"; | ||
| The returned function is powered by a zlib class instance behind the scenes. | ||
| let chunk1 = deflate(data); | ||
| // Buffer(17) [120, 156, 50, 52, 50, 54, 49, 53, 51, 183, 176, 4, 0, 0, 0, 255, 255] | ||
| // first chunk of data sets up the shared context | ||
| ### instance(data[, flag]) | ||
| let chunk2 = deflate(data); | ||
| // Buffer(9) [50, 132, 49, 0, 0, 0, 0, 255, 255] | ||
| // shared context kicks in | ||
| - `data` (Buffer-compatible): The data to be compressed or decompressed | ||
| - `flag` (Integer): A flush flag to override the default | ||
| - Returns (Buffer): A Buffer of data | ||
| let chunk3 = deflate(data); | ||
| // Buffer(8) [130, 51, 0, 0, 0, 0, 255, 255] | ||
| // and continues to apply to all subsequent chunks | ||
| let decoded1 = inflate(chunk1); | ||
| console.log(decoded1.toString()); // "123456789" | ||
| let decoded2 = inflate(chunk2); | ||
| console.log(decoded2.toString()); // "123456789" | ||
| let decoded3 = inflate(chunk3); | ||
| console.log(decoded3.toString()); // "123456789" | ||
| ``` | ||
| This package is essentially a function that returns a compressor or a decompressor function powered by zlib behind the scenes. | ||
| All major zlib classes are supported: | ||
@@ -68,16 +50,45 @@ | ||
| Compressors and decompressors accept any Buffer-compatible input (Buffer, TypedArray, DataView, ArrayBuffer, string) and return a Buffer. | ||
| ## Usage Examples | ||
| Usage is similar to any other synchronous compression library, compress a chunk, then decompress it elsewhere, except that the functions keep track of its compression state and sliding window contexts. | ||
| ```js | ||
| let data = gzip("wefwefwef"); | ||
| console.log(data); // compressed buffer | ||
| let zlib = require("fast-zlib"); | ||
| let decompressed = unzip(data); | ||
| console.log(decompressed); // decompressed buffer | ||
| console.log(decompressed.toString()); // wefwefwef | ||
| let deflate = zlib("deflate"); // create a deflator | ||
| let inflate = zlib("inflate"); // create an inflator | ||
| let data = "123456789"; | ||
| let chunk1 = deflate(data); | ||
| // Buffer(17) [120, 156, 50, 52, 50, 54, 49, 53, 51, 183, 176, 4, 0, 0, 0, 255, 255] | ||
| // first chunk of data is fully processed | ||
| let chunk2 = deflate(data); | ||
| // Buffer(9) [50, 132, 49, 0, 0, 0, 0, 255, 255] | ||
| // reusable patterns from previous compression are referenced | ||
| let chunk3 = deflate(data); | ||
| // Buffer(8) [130, 51, 0, 0, 0, 0, 255, 255] | ||
| // and continues to be applied to all subsequent chunks | ||
| inflate(chunk1).toString(); // "123456789" | ||
| inflate(chunk2).toString(); // "123456789" | ||
| inflate(chunk3).toString(); // "123456789" | ||
| ``` | ||
| Each zlib class can be passed an options object as per [zlib's documentation](https://nodejs.org/docs/latest-v12.x/api/zlib.html). | ||
| Decompression must be done in exactly the same order as compression because each chunk sequentially complements the previous and the next. Attempting to decode a chunk out of order will throw an error and reset the decompressor so it has to either restart from the beginning or you will have to destroy both and create a new compressor/decompressor pair. | ||
| ```js | ||
| let chunk1 = deflate(data); | ||
| let chunk2 = deflate(data); | ||
| inflate(chunk2); // error | ||
| inflate(chunk1); // works | ||
| inflate(chunk2); // works | ||
| ``` | ||
| Each zlib class can be passed an options object as per zlib's documentation. | ||
| ```js | ||
| let deflateRaw = zlib("deflateRaw", { | ||
@@ -95,3 +106,3 @@ chunkSize: 128 * 1024, | ||
| ```js | ||
| // all of zlib's constants are accessible from this package | ||
| // all of zlib's constants are accessible | ||
| let deflate = zlib("deflate", { | ||
@@ -102,41 +113,14 @@ flush: zlib.constants.Z_NO_FLUSH // set default flag to Z_NO_FLUSH | ||
| deflate("123"); | ||
| deflate("123"); // add data without processing | ||
| deflate("456"); | ||
| deflate("789"); | ||
| let data = deflate("hij", zlib.constants.Z_SYNC_FLUSH); | ||
| let data = deflate("hij", zlib.constants.Z_SYNC_FLUSH); // process all data added so far and return it | ||
| let result = inflate(data); | ||
| console.log(result.toString()); // 123456789hij | ||
| inflate(data).toString(); // 123456789hij | ||
| ``` | ||
| The function's internal zlib instance is also exposed via a .zlib property for advanced usage. | ||
| Flush flags can be used to achieve fine control over the compression process and even create checkpoints from where decompression can resume | ||
| ```js | ||
| let deflate = zlib("deflate"); | ||
| let deflate = zlib("inflate"); | ||
| deflate("789", zlib.constants.Z_NO_FLUSH); | ||
| // here deflate.zlib is an instance of zlib.createDeflate() and we can use its internal methods | ||
| deflate.zlib.flush(); | ||
| let data = deflate.zlib.read(); | ||
| console.log(inflate(data).toString()); // 789 | ||
| ``` | ||
| In shared context, decompression must be done in exactly the same order as compression because each chunk sequentially complements the previous and the next. Attempting to decode a chunk out of order will throw an error and reset the decompressor so it has to either restart from the beginning or you will have to destroy both and create a new compressor/decompressor pair. | ||
| ```js | ||
| let chunk1 = deflate(data); | ||
| let chunk2 = deflate(data); | ||
| inflate(chunk2); // error | ||
| inflate(chunk1); // works | ||
| inflate(chunk2); // works | ||
| ``` | ||
| Flush flags can be used to achieve fine control over the process and even create checkpoints from where decompression can resume | ||
| ```js | ||
| let deflate = zlib("deflateRaw"); | ||
@@ -156,12 +140,12 @@ let inflate = zlib("inflateRaw"); | ||
| console.log(inflate(data).toString()) // 123456789 | ||
| console.log(inflate(data2).toString()) // abc789 | ||
| console.log(inflate(data3).toString()) // zyx789 | ||
| inflate(data).toString(); // 123456789 | ||
| inflate(data2).toString(); // abc789 | ||
| inflate(data3).toString(); // zyx789 | ||
| // we can restart the decompression sequence from a Z_FULL_FLUSH block at any time | ||
| console.log(inflate(data2).toString()) // abc789 | ||
| console.log(inflate(data3).toString()) // xyz789 | ||
| inflate(data2).toString(); // abc789 | ||
| inflate(data3).toString(); // xyz789 | ||
| ``` | ||
| Not all classes support the same flags and there might be small differences in behavior between deflate, gzip and brotli. For example brotli uses slightly different flush flags compared to deflate and gzip. Instead of `Z_NO_FLUSH` and `Z_SYNC_FLUSH`, its flags are `BROTLI_OPERATION_PROCESS` and `BROTLI_OPERATION_FLUSH`. Check zlib's documentation for more details about how each class works. | ||
| Not all classes support the same flags and there might be small differences in behavior between deflate, gzip and brotli. For example brotli uses different flush flags compared to deflate and gzip. Instead of `Z_NO_FLUSH` and `Z_SYNC_FLUSH`, its flags are `BROTLI_OPERATION_PROCESS` and `BROTLI_OPERATION_FLUSH`. Check zlib's documentation for more details about how each class works. | ||
@@ -174,3 +158,3 @@ ```js | ||
| let compressed = compress("abc"); | ||
| console.log(decompress(compressed).toString()); // abc | ||
| decompress(compressed).toString(); // abc | ||
@@ -180,17 +164,4 @@ compress("123",zlib.constants.BROTLI_OPERATION_PROCESS); | ||
| let data = compress("789",zlib.constants.BROTLI_OPERATION_FLUSH); | ||
| console.log(decompress(data).toString()); // 123456789 | ||
| ``` | ||
| When working with streams where fragmentation can occur (such as TCP streams) its a good idea to watch for zlib's delimiter and join chunks together. Decompression will still work with incomplete chunks but will return incomplete data that you will need to process yourself. | ||
| ```js | ||
| stream.on("data", chunk => { | ||
| if(chunk.length >= 4 && chunk.readUInt32BE(chunk.length - 4) === 0xffff) { // check if the chunk ends with 0,0,255,255 | ||
| let data = inflate(chunk, zlib.constants.Z_SYNC_FLUSH); // if it does, process it and continue | ||
| console.log(data.toString()) | ||
| } else { | ||
| inflate(chunk, zlib.constants.Z_NO_FLUSH); // otherwise add it to the internal buffer and wait for the next chunk | ||
| return; | ||
| } | ||
| }); | ||
| decompress(data).toString(); // 123456789 | ||
| ``` | ||
@@ -200,8 +171,29 @@ | ||
| soon | ||
| Tested on Node.js v12.16.1 running on an i5 7300HQ 2.5ghz with default zlib options | ||
| ## Unsafe Mode | ||
| Deflate performance on randomized json messages of various sizes | ||
| This package contains an additional `Z_SYNC_FLUSH_UNSAFE` experimental flag for maximum performance, but it can cause some issues if not used carefully. It cannot be used with deflate's Z_FULL_FLUSH, it does not append zlib's signature block delimiter (0,0,255,255), only accepts a Buffer as input, and reuses existing buffers when possible. | ||
| | Library | \~ 0.03kb | \~ 0.5kb | \~ 11kb | | ||
| |---------------|---------|--------|-------| | ||
| | zlib (stream) | 12839 op/s | 7441 op/s | 978 op/s | | ||
| | pako (stream) | 23071 op/s | 8961 op/s | 638 op/s | | ||
| | minizlib | 38939 op/s | 14933 op/s | 1131 op/s | | ||
| | fast-zlib | 61899 op/s | 16533 op/s | 1125 op/s | | ||
| Inflate performance on the same messages | ||
| | Library | \~ 0.03kb | \~ 0.5kb | \~ 11kb | | ||
| |---------------|---------|--------|-------| | ||
| | zlib (stream) | 13579 op/s | 10378 op/s | 3276 op/s | | ||
| | pako (stream) | 44077 op/s | 25278 op/s | 4159 op/s | | ||
| | minizlib | 68383 op/s | 39486 op/s | 7776 op/s | | ||
| | zlib-sync | 126418 op/s | 46206 op/s | 7384 op/s | | ||
| | fast-zlib | 138365 op/s | 53413 op/s | 8097 op/s | | ||
| More benchmarks can be found at [zlib-benchmark](https://github.com/timotejroiko/zlib-benchmark) | ||
| ## Experimental Unsafe Mode | ||
| This package contains an additional `Z_SYNC_FLUSH_UNSAFE` experimental flag for maximum performance, but it can cause issues if not used carefully. It cannot be interchanged with Z_FULL_FLUSH, it does not append zlib's signature block delimiter (0,0,255,255), only accepts a Buffer as input, and reuses existing buffers when possible. | ||
| ```js | ||
@@ -208,0 +200,0 @@ let deflate = zlib("deflate"); |
New alerts
License Policy Violation
LicenseThis package is not allowed per your license policy. Review the package's license to ensure compliance.
Found 1 instance in 1 package
Fixed alerts
License Policy Violation
LicenseThis package is not allowed per your license policy. Review the package's license to ensure compliance.
Found 1 instance in 1 package
Worsened metrics
- Total package byte prevSize
- decreased by-1.33%
12114
- Number of lines in readme file
- decreased by-3.81%
202
No dependency changes