@devgrid/messagepack

@devgrid/messagepack

A high-performance MessagePack implementation for Node.js with TypeScript support, custom type extensions, and streaming capabilities.

Features

• 🚀 Full TypeScript support
• 📦 Minimal dependencies
• ⚡ High performance serialization/deserialization
• 🔄 Custom type extensions
• 💾 Built-in support for JavaScript native types
• 🛡️ Comprehensive error handling
• 🔍 Streaming support via SmartBuffer

Installation

npm install @devgrid/messagepack
# or
yarn add @devgrid/messagepack

Basic Usage

import { encode, decode } from '@devgrid/messagepack';

// Encoding
const data = {
  name: "John",
  age: 30,
  hobbies: ["reading", "gaming"]
};
const encoded = encode(data);

// Decoding
const decoded = decode(encoded);

Supported Types

Built-in Types

// Numbers (integers and floats)
encode(42);
encode(3.14);

// Strings
encode("Hello World");

// Booleans
encode(true);
encode(false);

// null
encode(null);

// Arrays
encode([1, 2, 3]);

// Objects
encode({ foo: "bar" });

// Buffer
encode(Buffer.from([1, 2, 3]));

Native JavaScript Types

import { serializer } from '@devgrid/messagepack';

// Date
const date = new Date();
const encodedDate = encode(date);

// Map
const map = new Map([["key", "value"]]);
const encodedMap = encode(map);

// Set
const set = new Set([1, 2, 3]);
const encodedSet = encode(set);

// RegExp
const regex = /pattern/g;
const encodedRegex = encode(regex);

// BigInt
const bigint = BigInt("9007199254740991");
const encodedBigInt = encode(bigint);

// Error objects
const error = new Error("Something went wrong");
const encodedError = encode(error);

Custom Type Extensions

You can register your own types for serialization:

import { Serializer } from '@devgrid/messagepack';

class User {
  constructor(public name: string, public age: number) {}
}

const customSerializer = new Serializer();

// Register custom type (use numbers 1-99 for user-defined types)
customSerializer.register(
  1, // type ID
  User, // constructor
  // Encoder
  (obj: User, buf: SmartBuffer) => {
    customSerializer.encode(obj.name, buf);
    customSerializer.encode(obj.age, buf);
  },
  // Decoder
  (buf: SmartBuffer) => {
    const name = customSerializer.decode(buf);
    const age = customSerializer.decode(buf);
    return new User(name, age);
  }
);

// Use custom serializer
const user = new User("John", 30);
const encoded = customSerializer.encode(user);
const decoded = customSerializer.decode(encoded);

Error Handling

The library includes built-in support for serializing standard JavaScript errors:

try {
  throw new TypeError("Invalid type");
} catch (error) {
  const encoded = encode(error);
  const decoded = decode(encoded);
  
  console.log(decoded instanceof TypeError); // true
  console.log(decoded.message); // "Invalid type"
  console.log(decoded.stack); // Preserved stack trace
}

Streaming Support

The library supports streaming via SmartBuffer:

import { SmartBuffer } from '@devgrid/smartbuffer';
import { serializer } from '@devgrid/messagepack';

// Create a buffer
const buf = new SmartBuffer();

// Encode multiple objects
serializer.encode({ type: "header" }, buf);
serializer.encode({ type: "data", content: "..." }, buf);
serializer.encode({ type: "footer" }, buf);

// Decode stream
while (buf.length > 0) {
  const result = serializer.decoder.tryDecode(buf);
  if (result) {
    console.log(result.value);
  } else {
    break; // Incomplete data
  }
}

Type Extension IDs

The library reserves specific type IDs for different purposes:

• 126: Standard Errors
• 125: Date
• 124: Map
• 123: Set
• 121: RegExp
• 120: BigInt
• 119: Long
• 110-118: Reserved for other system types
• 100-109: Reserved for network types
• 1-99: Available for user-defined types

Performance Considerations

• Uses SmartBuffer for efficient buffer operations
• Optimized encoding/decoding paths for common types
• Minimal allocations during serialization
• Efficient handling of binary data

API Reference

Main Functions

encode(value: any): Buffer
decode(buffer: Buffer | SmartBuffer): any
tryDecode(buffer: SmartBuffer): { value: any, bytesConsumed: number } | null

Serializer Class

class Serializer {
  constructor(initialCapacity?: number);
  
  register(type: number, constructor: any, encode: EncodeFunction, decode: DecodeFunction): this;
  registerEncoder(type: number, check: CheckFunction, encode: EncodeFunction): this;
  registerDecoder(type: number, decode: DecodeFunction): this;
  
  encode(value: any, buffer?: SmartBuffer): SmartBuffer;
  decode(buffer: Buffer | SmartBuffer): any;
}

License

MIT

Credits

Built with SmartBuffer for efficient buffer operations.

Browser Support

While this library is primarily designed for Node.js, it can be used in browser environments with proper bundling setup.

Using with Bundlers

// webpack/rollup/esbuild will handle the Buffer polyfill
import { encode, decode } from '@devgrid/messagepack';

Bundler Configuration

Webpack

// webpack.config.js
module.exports = {
  resolve: {
    fallback: {
      "buffer": require.resolve("buffer/"),
      "long": require.resolve("long")
    }
  },
  plugins: [
    new webpack.ProvidePlugin({
      Buffer: ['buffer', 'Buffer']
    })
  ]
};

Rollup

// rollup.config.js
import nodePolyfills from 'rollup-plugin-node-polyfills';

export default {
  plugins: [
    nodePolyfills()
  ]
};

Browser-Specific Considerations

• Requires Buffer polyfill (automatically handled by most bundlers)
• Long.js is required for 64-bit integer support
• Performance may vary compared to Node.js environment
• Streaming capabilities might be limited in browser context
• Consider using smaller alternatives for browser-only projects

Browser Usage Example

<!-- Include required polyfills -->
<script src="https://cdn.jsdelivr.net/npm/long/dist/long.js"></script>
<script src="https://cdn.jsdelivr.net/npm/buffer@6.0.3/index.min.js"></script>

<!-- Include bundled messagepack -->
<script src="path/to/bundled/messagepack.js"></script>

<script>
  const data = {
    name: "John",
    numbers: [1, 2, 3],
    timestamp: new Date()
  };

  // Encode data
  const encoded = messagepack.encode(data);

  // Decode data
  const decoded = messagepack.decode(encoded);
</script>

Browser Bundle Size Optimization

To minimize bundle size when using in browsers:

• Use tree-shaking capable bundler
• Import only required functionality:

import { encode, decode } from '@devgrid/messagepack';
// Instead of
import * as messagepack from '@devgrid/messagepack';

• Consider using separate bundles for modern and legacy browsers
• Implement custom serializers only for types you need