Security News
JSR Working Group Kicks Off with Ambitious Roadmap and Plans for Open Governance
At its inaugural meeting, the JSR Working Group outlined plans for an open governance model and a roadmap to enhance JavaScript package management.
@msgpack/msgpack
Advanced tools
@msgpack/msgpack is an npm package that provides functionality for encoding and decoding data using the MessagePack format. MessagePack is a binary serialization format that is more efficient than JSON in terms of both size and speed. This package allows you to serialize JavaScript objects into a compact binary format and deserialize them back into JavaScript objects.
Encoding Data
This feature allows you to encode a JavaScript object into a MessagePack binary format. The `encode` function takes a JavaScript object and returns a Buffer containing the binary data.
const msgpack = require('@msgpack/msgpack');
const data = { foo: 'bar', num: 42 };
const encoded = msgpack.encode(data);
console.log(encoded);
Decoding Data
This feature allows you to decode a MessagePack binary format back into a JavaScript object. The `decode` function takes a Buffer containing the binary data and returns the original JavaScript object.
const msgpack = require('@msgpack/msgpack');
const encoded = Buffer.from([0x82, 0xa3, 0x66, 0x6f, 0x6f, 0xa3, 0x62, 0x61, 0x72, 0xa3, 0x6e, 0x75, 0x6d, 0x2a]);
const decoded = msgpack.decode(encoded);
console.log(decoded);
Custom Extension Types
This feature allows you to define custom extension types for encoding and decoding. You can use `addExtPacker` to specify how to serialize a custom type and `addExtUnpacker` to specify how to deserialize it.
const msgpack = require('@msgpack/msgpack');
class MyType {
constructor(value) {
this.value = value;
}
}
msgpack.addExtPacker(0x01, MyType, (obj) => msgpack.encode(obj.value));
msgpack.addExtUnpacker(0x01, (data) => new MyType(msgpack.decode(data)));
const myObj = new MyType('custom data');
const encoded = msgpack.encode(myObj);
const decoded = msgpack.decode(encoded);
console.log(decoded);
msgpack-lite is another npm package for MessagePack serialization. It is designed to be lightweight and fast, with a focus on performance. Compared to @msgpack/msgpack, msgpack-lite may have fewer features but is optimized for speed and small bundle size.
notepack.io is a fast and small MessagePack implementation for JavaScript. It is designed to be highly efficient and is often used in performance-critical applications. Compared to @msgpack/msgpack, notepack.io is more focused on performance and may offer better speed at the cost of some additional features.
msgpack5 is a pure JavaScript implementation of the MessagePack format. It supports both encoding and decoding, as well as custom extension types. Compared to @msgpack/msgpack, msgpack5 offers similar functionality but may have different performance characteristics and API design.
This is a JavaScript/ECMA-262 implementation of MessagePack, an efficient binary serilization format:
This library is a universal JavaScript, meaning it is compatible with all the major browsers and NodeJS. In addition, because it is implemented in TypeScript, type definition files (d.ts
) are always up-to-date and bundled in the distribution.
Note that this is the second version of MessagePack for JavaScript. The first version, which was implemented in ES5 and was never released to npmjs.com, is tagged as classic.
import { deepStrictEqual } from "assert";
import { encode, decode } from "@msgpack/msgpack";
const object = {
nil: null,
integer: 1,
float: Math.PI,
string: "Hello, world!",
binary: Uint8Array.from([1, 2, 3]),
array: [10, 20, 30],
map: { foo: "bar" },
timestampExt: new Date(),
};
const encoded: Uint8Array = encode(object);
deepStrictEqual(decode(encoded), object);
encode(data: unknown, options?: EncoderOptions): Uint8Array
decode(buffer: ArrayLike<number> | BufferSource, options?: DecoderOptions): unknown
decodeMulti(buffer: ArrayLike<number> | BufferSource, options?: DecoderOptions): Generator<unknown, void, unknown>
decodeAsync(stream: ReadableStreamLike<ArrayLike<number> | BufferSource>, options?: DecoderOptions): Promise<unknown>
decodeArrayStream(stream: ReadableStreamLike<ArrayLike<number> | BufferSource>, options?: DecoderOptions): AsyncIterable<unknown>
decodeMultiStream(stream: ReadableStreamLike<ArrayLike<number> | BufferSource>, options?: DecoderOptions): AsyncIterable<unknown>
This library is published to npmjs.com
as @msgpack/msgpack.
npm install @msgpack/msgpack
encode(data: unknown, options?: EncoderOptions): Uint8Array
It encodes data
into a single MessagePack-encoded object, and returns a byte array as Uint8Array
. It throws errors if data
is, or includes, a non-serializable object such as a function
or a symbol
.
for example:
import { encode } from "@msgpack/msgpack";
const encoded: Uint8Array = encode({ foo: "bar" });
console.log(encoded);
If you'd like to convert an uint8array
to a NodeJS Buffer
, use Buffer.from(arrayBuffer, offset, length)
in order not to copy the underlying ArrayBuffer
, while Buffer.from(uint8array)
copies it:
import { encode } from "@msgpack/msgpack";
const encoded: Uint8Array = encode({ foo: "bar" });
// `buffer` refers the same ArrayBuffer as `encoded`.
const buffer: Buffer = Buffer.from(encoded.buffer, encoded.byteOffset, encoded.byteLength);
console.log(buffer);
EncoderOptions
Name | Type | Default |
---|---|---|
extensionCodec | ExtensionCodec | ExtensionCodec.defaultCodec |
maxDepth | number | 100 |
initialBufferSize | number | 2048 |
sortKeys | boolean | false |
forceFloat32 | boolean | false |
forceIntegerToFloat | boolean | false |
ignoreUndefined | boolean | false |
context | user-defined | - |
decode(buffer: ArrayLike<number> | BufferSource, options?: DecoderOptions): unknown
It decodes buffer
that includes a MessagePack-encoded object, and returns the decoded object typed unknown
.
buffer
must be an array of bytes, which is typically Uint8Array
or ArrayBuffer
. BufferSource
is defined as ArrayBuffer | ArrayBufferView
.
The buffer
must include a single encoded object. If the buffer
includes extra bytes after an object or the buffer
is empty, it throws RangeError
. To decode buffer
that includes multiple encoded objects, use decodeMulti()
or decodeMultiStream()
(recommended) instead.
for example:
import { decode } from "@msgpack/msgpack";
const encoded: Uint8Array;
const object = decode(encoded);
console.log(object);
NodeJS Buffer
is also acceptable because it is a subclass of Uint8Array
.
DecoderOptions
Name | Type | Default |
---|---|---|
extensionCodec | ExtensionCodec | ExtensionCodec.defaultCodec |
maxStrLength | number | 4_294_967_295 (UINT32_MAX) |
maxBinLength | number | 4_294_967_295 (UINT32_MAX) |
maxArrayLength | number | 4_294_967_295 (UINT32_MAX) |
maxMapLength | number | 4_294_967_295 (UINT32_MAX) |
maxExtLength | number | 4_294_967_295 (UINT32_MAX) |
context | user-defined | - |
You can use max${Type}Length
to limit the length of each type decoded.
decodeMulti(buffer: ArrayLike<number> | BufferSource, options?: DecoderOptions): Generator<unknown, void, unknown>
It decodes buffer
that includes multiple MessagePack-encoded objects, and returns decoded objects as a generator. See also decodeMultiStream()
, which is an asynchronous variant of this function.
This function is not recommended to decode a MessagePack binary via I/O stream including sockets because it's synchronous. Instead, decodeMultiStream()
decodes a binary stream asynchronously, typically spending less CPU and memory.
for example:
import { decode } from "@msgpack/msgpack";
const encoded: Uint8Array;
for (const object of decodeMulti(encoded)) {
console.log(object);
}
decodeAsync(stream: ReadableStreamLike<ArrayLike<number> | BufferSource>, options?: DecoderOptions): Promise<unknown>
It decodes stream
, where ReadableStreamLike<T>
is defined as ReadableStream<T> | AsyncIterable<T>
, in an async iterable of byte arrays, and returns decoded object as unknown
type, wrapped in Promise
.
This function works asynchronously, and might CPU resources more efficiently compared with synchronous decode()
, because it doesn't wait for the completion of downloading.
This function is designed to work with whatwg fetch()
like this:
import { decodeAsync } from "@msgpack/msgpack";
const MSGPACK_TYPE = "application/x-msgpack";
const response = await fetch(url);
const contentType = response.headers.get("Content-Type");
if (contentType && contentType.startsWith(MSGPACK_TYPE) && response.body != null) {
const object = await decodeAsync(response.body);
// do something with object
} else { /* handle errors */ }
decodeArrayStream(stream: ReadableStreamLike<ArrayLike<number> | BufferSource>, options?: DecoderOptions): AsyncIterable<unknown>
It is alike to decodeAsync()
, but only accepts a stream
that includes an array of items, and emits a decoded item one by one.
for example:
import { decodeArrayStream } from "@msgpack/msgpack";
const stream: AsyncIterator<Uint8Array>;
// in an async function:
for await (const item of decodeArrayStream(stream)) {
console.log(item);
}
decodeMultiStream(stream: ReadableStreamLike<ArrayLike<number> | BufferSource>, options?: DecoderOptions): AsyncIterable<unknown>
It is alike to decodeAsync()
and decodeArrayStream()
, but the input stream
must consist of multiple MessagePack-encoded items. This is an asynchronous variant for decodeMulti()
.
In other words, it could decode an unlimited stream and emits a decoded item one by one.
for example:
import { decodeMultiStream } from "@msgpack/msgpack";
const stream: AsyncIterator<Uint8Array>;
// in an async function:
for await (const item of decodeMultiStream(stream)) {
console.log(item);
}
This function is available since v2.4.0; previously it was called as decodeStream()
.
Encoder
and Decoder
classes are provided to have better performance by reusing instances:
import { deepStrictEqual } from "assert";
import { Encoder, Decoder } from "@msgpack/msgpack";
const encoder = new Encoder();
const decoder = new Decoder();
const encoded: Uint8Array = encoder.encode(object);
deepStrictEqual(decoder.decode(encoded), object);
According to our benchmark, reusing Encoder
instance is about 20% faster
than encode()
function, and reusing Decoder
instance is about 2% faster
than decode()
function. Note that the result should vary in environments
and data structure.
Encoder
and Decoder
take the same options as encode()
and decode()
respectively.
To handle MessagePack Extension Types, this library provides ExtensionCodec
class.
This is an example to setup custom extension types that handles Map
and Set
classes in TypeScript:
import { encode, decode, ExtensionCodec } from "@msgpack/msgpack";
const extensionCodec = new ExtensionCodec();
// Set<T>
const SET_EXT_TYPE = 0 // Any in 0-127
extensionCodec.register({
type: SET_EXT_TYPE,
encode: (object: unknown): Uint8Array | null => {
if (object instanceof Set) {
return encode([...object]);
} else {
return null;
}
},
decode: (data: Uint8Array) => {
const array = decode(data) as Array<unknown>;
return new Set(array);
},
});
// Map<T>
const MAP_EXT_TYPE = 1; // Any in 0-127
extensionCodec.register({
type: MAP_EXT_TYPE,
encode: (object: unknown): Uint8Array => {
if (object instanceof Map) {
return encode([...object]);
} else {
return null;
}
},
decode: (data: Uint8Array) => {
const array = decode(data) as Array<[unknown, unknown]>;
return new Map(array);
},
});
const encoded = encode([new Set<any>(), new Map<any, any>()], { extensionCodec });
const decoded = decode(encoded, { extensionCodec });
Not that extension types for custom objects must be [0, 127]
, while [-1, -128]
is reserved for MessagePack itself.
When you use an extension codec, it might be necessary to have encoding/decoding state to keep track of which objects got encoded/re-created. To do this, pass a context
to the EncoderOptions
and DecoderOptions
:
import { encode, decode, ExtensionCodec } from "@msgpack/msgpack";
class MyContext {
track(object: any) { /*...*/ }
}
class MyType { /* ... */ }
const extensionCodec = new ExtensionCodec<MyContext>();
// MyType
const MYTYPE_EXT_TYPE = 0 // Any in 0-127
extensionCodec.register({
type: MYTYPE_EXT_TYPE,
encode: (object, context) => {
if (object instanceof MyType) {
context.track(object); // <-- like this
return encode(object.toJSON(), { extensionCodec, context });
} else {
return null;
}
},
decode: (data, extType, context) => {
const decoded = decode(data, { extensionCodec, context });
const my = new MyType(decoded);
context.track(my); // <-- and like this
return my;
},
});
// and later
import { encode, decode } from "@msgpack/msgpack";
const context = new MyContext();
const encoded = = encode({myType: new MyType<any>()}, { extensionCodec, context });
const decoded = decode(encoded, { extensionCodec, context });
This library does not handle BigInt by default, but you can handle it with ExtensionCodec
like this:
import { deepStrictEqual } from "assert";
import { encode, decode, ExtensionCodec } from "@msgpack/msgpack";
const BIGINT_EXT_TYPE = 0; // Any in 0-127
const extensionCodec = new ExtensionCodec();
extensionCodec.register({
type: BIGINT_EXT_TYPE,
encode: (input: unknown) => {
if (typeof input === "bigint") {
if (input <= Number.MAX_SAFE_INTEGER && input >= Number.MIN_SAFE_INTEGER) {
return encode(parseInt(input.toString(), 10));
} else {
return encode(input.toString());
}
} else {
return null;
}
},
decode: (data: Uint8Array) => {
return BigInt(decode(data));
},
});
const value = BigInt(Number.MAX_SAFE_INTEGER) + BigInt(1);
const encoded: = encode(value, { extensionCodec });
deepStrictEqual(decode(encoded, { extensionCodec }), value);
There is a proposal for a new date/time representations in JavaScript:
This library maps Date
to the MessagePack timestamp extension by default, but you can re-map the temporal module (or Temporal Polyfill) to the timestamp extension like this:
import { Instant } from "@std-proposal/temporal";
import { deepStrictEqual } from "assert";
import {
encode,
decode,
ExtensionCodec,
EXT_TIMESTAMP,
encodeTimeSpecToTimestamp,
decodeTimestampToTimeSpec,
} from "@msgpack/msgpack";
const extensionCodec = new ExtensionCodec();
extensionCodec.register({
type: EXT_TIMESTAMP, // override the default behavior!
encode: (input: any) => {
if (input instanceof Instant) {
const sec = input.seconds;
const nsec = Number(input.nanoseconds - BigInt(sec) * BigInt(1e9));
return encodeTimeSpecToTimestamp({ sec, nsec });
} else {
return null;
}
},
decode: (data: Uint8Array) => {
const timeSpec = decodeTimestampToTimeSpec(data);
const sec = BigInt(timeSpec.sec);
const nsec = BigInt(timeSpec.nsec);
return Instant.fromEpochNanoseconds(sec * BigInt(1e9) + nsec);
},
});
const instant = Instant.fromEpochMilliseconds(Date.now());
const encoded = encode(instant, { extensionCodec });
const decoded = decode(encoded, { extensionCodec });
deepStrictEqual(decoded, instant);
This will become default in this library with major-version increment, if the temporal module is standardized.
Blob
is a binary data container provided by browsers. To read its contents, you can use Blob#arrayBuffer()
or Blob#stream()
. Blob#stream()
is recommended if your target platform support it. This is because streaming
decode should be faster for large objects. In both ways, you need to use
asynchronous API.
async function decodeFromBlob(blob: Blob): unknown {
if (blob.stream) {
// Blob#stream(): ReadableStream<Uint8Array> (recommended)
return await decodeAsync(blob.stream());
} else {
// Blob#arrayBuffer(): Promise<ArrayBuffer> (if stream() is not available)
return decode(await blob.arrayBuffer());
}
}
This library is compatible with the "August 2017" revision of MessagePack specification at the point where timestamp ext was added:
The living specification is here:
https://github.com/msgpack/msgpack
Note that as of June 2019 there're no official "version" on the MessagePack specification. See https://github.com/msgpack/msgpack/issues/195 for the discussions.
The following table shows how JavaScript values are mapped to MessagePack formats and vice versa.
Source Value | MessagePack Format | Value Decoded |
---|---|---|
null, undefined | nil | null (*1) |
boolean (true, false) | bool family | boolean (true, false) |
number (53-bit int) | int family | number (53-bit int) |
number (64-bit float) | float family | number (64-bit float) |
string | str family | string |
ArrayBufferView | bin family | Uint8Array (*2) |
Array | array family | Array |
Object | map family | Object (*3) |
Date | timestamp ext family | Date (*4) |
null
and undefined
are mapped to nil
(0xC0
) type, and are decoded into null
ArrayBufferView
s including NodeJS's Buffer
are mapped to bin
family, and are decoded into Uint8Array
Object
, it is regarded as Record<string, unknown>
in terms of TypeScriptDate
. This behavior can be overridden by registering -1
for the extension codec.This is a universal JavaScript library that supports major browsers and NodeJS.
ES2018 standard library used in this library can be polyfilled with core-js.
If you support IE11, import core-js
in your application entrypoints, as this library does in testing for browsers.
NodeJS v10 is required, but NodeJS v12 or later is recommended because it includes the V8 feature of Improving DataView performance in V8.
NodeJS before v10 will work by importing @msgpack/msgpack/dist.es5+umd/msgpack
.
This module requires type definitions of AsyncIterator
, SourceBuffer
, whatwg streams, and so on. They are provided by "lib": ["ES2021", "DOM"]
in tsconfig.json
.
Regarding the TypeScript compiler version, only the latest TypeScript is tested in development.
Run-time performance is not the only reason to use MessagePack, but it's important to choose MessagePack libraries, so a benchmark suite is provided to monitor the performance of this library.
V8's built-in JSON has been improved for years, esp. JSON.parse()
is significantly improved in V8/7.6, it is the fastest deserializer as of 2019, as the benchmark result bellow suggests.
However, MessagePack can handles binary data effectively, actual performance depends on situations. You'd better take benchmark on your own use-case if performance matters.
Benchmark on NodeJS/v18.1.0 (V8/10.1)
operation | op | ms | op/s |
---|---|---|---|
buf = Buffer.from(JSON.stringify(obj)); | 902100 | 5000 | 180420 |
obj = JSON.parse(buf.toString("utf-8")); | 898700 | 5000 | 179740 |
buf = require("msgpack-lite").encode(obj); | 411000 | 5000 | 82200 |
obj = require("msgpack-lite").decode(buf); | 246200 | 5001 | 49230 |
buf = require("@msgpack/msgpack").encode(obj); | 843300 | 5000 | 168660 |
obj = require("@msgpack/msgpack").decode(buf); | 489300 | 5000 | 97860 |
buf = /* @msgpack/msgpack */ encoder.encode(obj); | 1154200 | 5000 | 230840 |
obj = /* @msgpack/msgpack */ decoder.decode(buf); | 448900 | 5000 | 89780 |
Note that JSON
cases use Buffer
to emulate I/O where a JavaScript string must be converted into a byte array encoded in UTF-8, whereas MessagePack modules deal with byte arrays.
The NPM package distributed in npmjs.com includes both ES2015+ and ES5 files:
dist/
is compiled into ES2019 with CommomJS, provided for NodeJS v10dist.es5+umd/
is compiled into ES5 with UMD
dist.es5+umd/msgpack.min.js
- the minified filedist.es5+umd/msgpack.js
- the non-minified filedist.es5+esm/
is compiled into ES5 with ES modules, provided for webpack-like bundlers and NodeJS's ESM-modeIf you use NodeJS and/or webpack, their module resolvers use the suitable one automatically.
This library is available via CDN:
<script crossorigin src="https://unpkg.com/@msgpack/msgpack"></script>
It loads MessagePack
module to the global object.
You can use this module on Deno.
See example/deno-*.ts
for examples.
deno.land/x
is not supported yet.
For simple testing:
npm run test
This library uses Travis CI.
test matrix:
target=es2019
/ target=es5
See test:* in package.json and .travis.yml for details.
# run tests on NodeJS, Chrome, and Firefox
make test-all
# edit the changelog
code CHANGELOG.md
# bump version
npm version patch|minor|major
# run the publishing task
make publish
npm run update-dependencies
Copyright 2019 The MessagePack community.
This software uses the ISC license:
https://opensource.org/licenses/ISC
See LICENSE for details.
3.0.0-beta1
https://github.com/msgpack/msgpack-javascript/compare/v2.8.0...v3.0.0-beta1
useBigInt64
to map JavaScript's BigInt to MessagePack's int64 and uint64 (#223)Encoder
and Decoder
, and describe the interfaces in README.md (#224):
new Encoder(options: EncoderOptions)
: it takes the same named-options as encode()
new Decoder(options: DecoderOptions)
: it takes the same named-options as decode()
FAQs
MessagePack for ECMA-262/JavaScript/TypeScript
The npm package @msgpack/msgpack receives a total of 0 weekly downloads. As such, @msgpack/msgpack popularity was classified as not popular.
We found that @msgpack/msgpack demonstrated a not healthy version release cadence and project activity because the last version was released a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
At its inaugural meeting, the JSR Working Group outlined plans for an open governance model and a roadmap to enhance JavaScript package management.
Security News
Research
An advanced npm supply chain attack is leveraging Ethereum smart contracts for decentralized, persistent malware control, evading traditional defenses.
Security News
Research
Attackers are impersonating Sindre Sorhus on npm with a fake 'chalk-node' package containing a malicious backdoor to compromise developers' projects.