msgpack-lite
Advanced tools
The msgpack-lite npm package is a library for encoding and decoding data in MessagePack format, which is a binary serialization format that is compact and efficient. It is often used for communication between different programming languages, as well as for storing and transmitting data in a space-efficient manner.
Encoding JavaScript objects to MessagePack format
This feature allows you to take a JavaScript object and encode it into a MessagePack binary buffer, which can then be sent over a network or saved to disk.
const msgpack = require('msgpack-lite');
const object = { hello: 'world' };
const encoded = msgpack.encode(object);Decoding MessagePack buffers to JavaScript objects
This feature allows you to take a MessagePack binary buffer and decode it into a JavaScript object, which can then be used within your application.
const msgpack = require('msgpack-lite');
const buffer = Buffer.from([0x81, 0xA5, 0x68, 0x65, 0x6C, 0x6C, 0x6F, 0xA5, 0x77, 0x6F, 0x72, 0x6C, 0x64]);
const decoded = msgpack.decode(buffer);Stream encoding and decoding
This feature provides streaming support for encoding and decoding operations, which is useful for handling large datasets or streaming data over a network.
const msgpack = require('msgpack-lite');
const fs = require('fs');
const encodeStream = msgpack.createEncodeStream();
const decodeStream = msgpack.createDecodeStream();
fs.createReadStream('input.msgpack')
.pipe(decodeStream)
.on('data', console.log);
fs.createWriteStream('output.msgpack')
.pipe(encodeStream);
encodeStream.write({ hello: 'world' });
encodeStream.end();msgpack5 is a MessagePack v5 implementation for Node.js and the browser. It offers a similar API to msgpack-lite but also includes additional features such as custom extension types and a more flexible API for handling different types of input and output.
protobufjs is a package that provides Protocol Buffers support, which is another binary serialization format similar to MessagePack. It is more schema-driven and typically requires pre-defined .proto files, whereas MessagePack works with dynamic data structures without the need for a schema.
bson is a package that allows you to serialize and deserialize data in BSON (Binary JSON) format, which is used by MongoDB. While BSON is similar to MessagePack in that it is a binary format, it is more closely related to JSON and includes additional data types like ObjectId and Date.
Fast Pure JavaScript MessagePack Encoder and Decoder
Online demo: http://kawanet.github.io/msgpack-lite/
var msgpack = require("msgpack-lite");
// encode from JS Object to MessagePack (Buffer)
var buffer = msgpack.encode({"foo": "bar"});
// decode from MessagePack (Buffer) to JS Object
var data = msgpack.decode(buffer); // => {"foo": "bar"}
// if encode/decode receives an invalid argument an error is thrown
var fs = require("fs");
var msgpack = require("msgpack-lite");
var writeStream = fs.createWriteStream("test.msp");
var encodeStream = msgpack.createEncodeStream();
encodeStream.pipe(writeStream);
// send multiple objects to stream
encodeStream.write({foo: "bar"});
encodeStream.write({baz: "qux"});
// call this once you're done writing to the stream.
encodeStream.end();
var fs = require("fs");
var msgpack = require("msgpack-lite");
var readStream = fs.createReadStream("test.msp");
var decodeStream = msgpack.createDecodeStream();
// show multiple objects decoded from stream
readStream.pipe(decodeStream).on("data", console.warn);
var msgpack = require("msgpack-lite");
// decode() accepts Buffer instance per default
msgpack.decode(Buffer([0x81, 0xA3, 0x66, 0x6F, 0x6F, 0xA3, 0x62, 0x61, 0x72]));
// decode() also accepts Array instance
msgpack.decode([0x81, 0xA3, 0x66, 0x6F, 0x6F, 0xA3, 0x62, 0x61, 0x72]);
// decode() accepts raw Uint8Array instance as well
msgpack.decode(new Uint8Array([0x81, 0xA3, 0x66, 0x6F, 0x6F, 0xA3, 0x62, 0x61, 0x72]));
A CLI tool bin/msgpack converts data stream from JSON to MessagePack and vice versa.
$ echo '{"foo": "bar"}' | ./bin/msgpack -Jm | od -tx1
0000000 81 a3 66 6f 6f a3 62 61 72
$ echo '{"foo": "bar"}' | ./bin/msgpack -Jm | ./bin/msgpack -Mj
{"foo":"bar"}
$ npm install --save msgpack-lite
Run tests on node.js:
$ make test
Run tests on browsers:
$ make test-browser-local
open the following url in a browser:
http://localhost:4000/__zuul
Browser version msgpack.min.js is also available. 50KB minified, 14KB gziped.
<!--[if lte IE 9]>
<script src="https://cdnjs.cloudflare.com/ajax/libs/es5-shim/4.1.10/es5-shim.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/json3/3.3.2/json3.min.js"></script>
<![endif]-->
<script src="https://rawgit.com/kawanet/msgpack-lite/master/dist/msgpack.min.js"></script>
<script>
// encode from JS Object to MessagePack (Uint8Array)
var buffer = msgpack.encode({foo: "bar"});
// decode from MessagePack (Uint8Array) to JS Object
var array = new Uint8Array([0x81, 0xA3, 0x66, 0x6F, 0x6F, 0xA3, 0x62, 0x61, 0x72]);
var data = msgpack.decode(array);
</script>
Step #1: write some code at first.
var msgpack = require("msgpack-lite");
var buffer = msgpack.encode({"foo": "bar"});
var data = msgpack.decode(buffer);
console.warn(data); // => {"foo": "bar"}
Proceed to the next steps if you prefer faster browserify compilation time.
Step #2: add browser property on package.json in your project. This refers the global msgpack object instead of including whole of msgpack-lite source code.
{
"dependencies": {
"msgpack-lite": "*"
},
"browser": {
"msgpack-lite": "msgpack-lite/global"
}
}
Step #3: compile it with browserify and uglifyjs.
browserify src/main.js -o tmp/main.browserify.js -s main
uglifyjs tmp/main.browserify.js -m -c -o js/main.min.js
cp node_modules/msgpack-lite/dist/msgpack.min.js js/msgpack.min.js
Step #4: load msgpack.min.js before your code.
<script src="js/msgpack.min.js"></script>
<script src="js/main.min.js"></script>
It is tested to have basic compatibility with other Node.js MessagePack modules below:
A benchmark tool lib/benchmark.js is available to compare encoding/decoding speed
(operation per second) with other MessagePack modules.
It counts operations of 1KB JSON document in 10 seconds.
$ npm install msgpack msgpack-js msgpack-js-v5 msgpack-unpack msgpack5 notepack
$ npm run benchmark 10
| operation | op | ms | op/s |
|---|---|---|---|
| buf = Buffer(JSON.stringify(obj)); | 1055200 | 10000 | 105520 |
| obj = JSON.parse(buf); | 863800 | 10000 | 86380 |
| buf = require("msgpack-lite").encode(obj); | 969100 | 10000 | 96910 |
| obj = require("msgpack-lite").decode(buf); | 600300 | 10000 | 60030 |
| buf = require("msgpack").pack(obj); | 503500 | 10001 | 50344 |
| obj = require("msgpack").unpack(buf); | 560200 | 10001 | 56014 |
| buf = Buffer(require("msgpack.codec").msgpack.pack(obj)); | 653500 | 10000 | 65349 |
| obj = require("msgpack.codec").msgpack.unpack(buf); | 367500 | 10001 | 36746 |
| buf = require("msgpack-js-v5").encode(obj); | 189500 | 10002 | 18946 |
| obj = require("msgpack-js-v5").decode(buf); | 408900 | 10000 | 40890 |
| buf = require("msgpack-js").encode(obj); | 189200 | 10000 | 18920 |
| obj = require("msgpack-js").decode(buf); | 375600 | 10002 | 37552 |
| buf = require("msgpack5")().encode(obj); | 110500 | 10009 | 11040 |
| obj = require("msgpack5")().decode(buf); | 165500 | 10000 | 16550 |
| buf = require("notepack")().encode(obj); | 847800 | 10000 | 84780 |
| obj = require("notepack")().decode(buf); | 599800 | 10000 | 59980 |
| obj = require("msgpack-unpack").decode(buf); | 48100 | 10002 | 4809 |
Streaming benchmark tool lib/benchmark-stream.js is also available.
It counts milliseconds for 1,000,000 operations of 30 bytes fluentd msgpack fragment.
This shows streaming encoding and decoding are super faster.
$ npm run benchmark-stream 2
| operation (1000000 x 2) | op | ms | op/s |
|---|---|---|---|
| stream.write(msgpack.encode(obj)); | 1000000 | 3027 | 330360 |
| stream.write(notepack.encode(obj)); | 1000000 | 2012 | 497017 |
| msgpack.Encoder().on("data",ondata).encode(obj); | 1000000 | 2956 | 338294 |
| msgpack.createEncodeStream().write(obj); | 1000000 | 1888 | 529661 |
| stream.write(msgpack.decode(buf)); | 1000000 | 2020 | 495049 |
| stream.write(notepack.decode(buf)); | 1000000 | 1794 | 557413 |
| msgpack.Decoder().on("data",ondata).decode(buf); | 1000000 | 2744 | 364431 |
| msgpack.createDecodeStream().write(buf); | 1000000 | 1341 | 745712 |
Test environment: msgpack-lite 0.1.14, Node v4.2.3, Intel(R) Xeon(R) CPU E5-2666 v3 @ 2.90GHz
The following table shows how JavaScript objects (value) will be mapped to MessagePack formats and vice versa.
| Source Value | MessagePack Format | Value Decoded |
|---|---|---|
| null, undefined | nil format family | null |
| Boolean (true, false) | bool format family | Boolean (true, false) |
| Number (32bit int) | int format family | Number (int or double) |
| Number (64bit double) | float format family | Number (double) |
| String | str format family | String |
| Buffer | bin format family | Buffer |
| Array | array format family | Array |
| Map | map format family | Map (if usemap=true) |
| Object (plain object) | map format family | Object (or Map if usemap=true) |
| Object (see below) | ext format family | Object (see below) |
Note that both null and undefined are mapped to nil 0xC1 type.
This means undefined value will be upgraded to null in other words.
The MessagePack specification allows 128 application-specific extension types. The library uses the following types to make round-trip conversion possible for JavaScript native objects.
| Type | Object | Type | Object |
|---|---|---|---|
| 0x00 | 0x10 | ||
| 0x01 | EvalError | 0x11 | Int8Array |
| 0x02 | RangeError | 0x12 | Uint8Array |
| 0x03 | ReferenceError | 0x13 | Int16Array |
| 0x04 | SyntaxError | 0x14 | Uint16Array |
| 0x05 | TypeError | 0x15 | Int32Array |
| 0x06 | URIError | 0x16 | Uint32Array |
| 0x07 | 0x17 | Float32Array | |
| 0x08 | 0x18 | Float64Array | |
| 0x09 | 0x19 | Uint8ClampedArray | |
| 0x0A | RegExp | 0x1A | ArrayBuffer |
| 0x0B | Boolean | 0x1B | Buffer |
| 0x0C | String | 0x1C | |
| 0x0D | Date | 0x1D | DataView |
| 0x0E | Error | 0x1E | |
| 0x0F | Number | 0x1F |
Other extension types are mapped to built-in ExtBuffer object.
Register a custom extension type number to serialize/deserialize your own class instances.
var msgpack = require("msgpack-lite");
var codec = msgpack.createCodec();
codec.addExtPacker(0x3F, MyVector, myVectorPacker);
codec.addExtUnpacker(0x3F, myVectorUnpacker);
var data = new MyVector(1, 2);
var encoded = msgpack.encode(data, {codec: codec});
var decoded = msgpack.decode(encoded, {codec: codec});
function MyVector(x, y) {
this.x = x;
this.y = y;
}
function myVectorPacker(vector) {
var array = [vector.x, vector.y];
return msgpack.encode(array); // return Buffer serialized
}
function myVectorUnpacker(buffer) {
var array = msgpack.decode(buffer);
return new MyVector(array[0], array[1]); // return Object deserialized
}
The first argument of addExtPacker and addExtUnpacker should be an integer within the range of 0 and 127 (0x0 and 0x7F). myClassPacker is a function that accepts an instance of MyClass, and should return a buffer representing that instance. myClassUnpacker is the opposite: it accepts a buffer and should return an instance of MyClass.
If you pass an array of functions to addExtPacker or addExtUnpacker, the value to be encoded/decoded will pass through each one in order. This allows you to do things like this:
codec.addExtPacker(0x00, Date, [Number, msgpack.encode]);
You can also pass the codec option to msgpack.Decoder(options), msgpack.Encoder(options), msgpack.createEncodeStream(options), and msgpack.createDecodeStream(options).
If you wish to modify the default built-in codec, you can access it at msgpack.codec.preset.
msgpack.createCodec() function accepts some options.
It does NOT have the preset extension types defined when no options given.
var codec = msgpack.createCodec();
preset: It has the preset extension types described above.
var codec = msgpack.createCodec({preset: true});
safe: It runs a validation of the value before writing it into buffer. This is the default behavior for some old browsers which do not support ArrayBuffer object.
var codec = msgpack.createCodec({safe: true});
useraw: It uses raw formats instead of bin and str.
var codec = msgpack.createCodec({useraw: true});
int64: It decodes msgpack's int64/uint64 formats with int64-buffer object.
var codec = msgpack.createCodec({int64: true});
binarraybuffer: It ties msgpack's bin format with ArrayBuffer object, instead of Buffer object.
var codec = msgpack.createCodec({binarraybuffer: true, preset: true});
uint8array: It returns Uint8Array object when encoding, instead of Buffer object.
var codec = msgpack.createCodec({uint8array: true});
usemap: Uses the global JavaScript Map type, if available, to unpack
MessagePack map elements.
var codec = msgpack.createCodec({usemap: true});
The compatibility mode respects for msgpack's old spec. Set true to useraw.
// default mode handles both str and bin formats individually
msgpack.encode("Aa"); // => <Buffer a2 41 61> (str format)
msgpack.encode(new Buffer([0x41, 0x61])); // => <Buffer c4 02 41 61> (bin format)
msgpack.decode(new Buffer([0xa2, 0x41, 0x61])); // => 'Aa' (String)
msgpack.decode(new Buffer([0xc4, 0x02, 0x41, 0x61])); // => <Buffer 41 61> (Buffer)
// compatibility mode handles only raw format both for String and Buffer
var options = {codec: msgpack.createCodec({useraw: true})};
msgpack.encode("Aa", options); // => <Buffer a2 41 61> (raw format)
msgpack.encode(new Buffer([0x41, 0x61]), options); // => <Buffer a2 41 61> (raw format)
msgpack.decode(new Buffer([0xa2, 0x41, 0x61]), options); // => <Buffer 41 61> (Buffer)
msgpack.decode(new Buffer([0xa2, 0x41, 0x61]), options).toString(); // => 'Aa' (String)
The MIT License (MIT)
Copyright (c) 2015-2016 Yusuke Kawasaki
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
FAQs
Fast Pure JavaScript MessagePack Encoder and Decoder
The npm package msgpack-lite receives a total of 2,180,965 weekly downloads. As such, msgpack-lite popularity was classified as popular.
We found that msgpack-lite demonstrated a not healthy version release cadence and project activity because the last version was released a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Research
Security News
Socket’s threat research team has detected six malicious npm packages typosquatting popular libraries to insert SSH backdoors.
Security News
MITRE's 2024 CWE Top 25 highlights critical software vulnerabilities like XSS, SQL Injection, and CSRF, reflecting shifts due to a refined ranking methodology.
Security News
In this segment of the Risky Business podcast, Feross Aboukhadijeh and Patrick Gray discuss the challenges of tracking malware discovered in open source softare.