token-types

What is token-types?

The token-types npm package provides a collection of token parsers for binary data types, facilitating the reading and interpretation of various data types from binary streams. This is particularly useful in applications dealing with multimedia files, where metadata and content need to be extracted or interpreted from binary formats.

What are token-types's main functionalities?

Reading integer values

This feature allows the reading of unsigned 32-bit integers from a buffer in little-endian format. It is useful for parsing low-level binary data structures.

const Token = require('token-types');

const uint32Token = Token.UINT32_LE;
const buffer = Buffer.from([0x01, 0x00, 0x00, 0x00]);
const value = uint32Token.get(buffer, 0);
console.log(value); // Outputs: 1

Reading string values

This feature enables the extraction of ASCII strings from binary data. It supports specifying the length of the string and the encoding, making it versatile for reading text data embedded in binary streams.

const Token = require('token-types');

const stringToken = new Token.StringType(5, 'ascii');
const buffer = Buffer.from('hello');
const value = stringToken.get(buffer, 0);
console.log(value); // Outputs: 'hello'

Other packages similar to token-types

binary-parser

binary-parser offers functionality to build a schema for parsing binary data, which is somewhat similar to token-types. However, binary-parser is more comprehensive in defining complex parsing schemas, making it suitable for applications that require detailed and structured binary data interpretation.

README

A primitive token library used to read from, and to write a node Buffer.

Tokens

node-strtok supports a wide variety of numerical tokens out of the box:

• UINT8
• UINT16_BE, UINT16_LE
• UINT24_BE, UINT24_LE
• UINT32_BE, UINT32_LE
• UINT64_BE, UINT64_LE*
• INT8
• INT16_BE, INT16_LE
• INT24_BE, INT24_LE
• INT32_BE, INT32_LE
• INT64_BE, UINT64_LE*

String types:

• Windows-1252
• ISO-8859-1

*) The 64-bit tokens are best effort based, since JavaScript limit value size to less than 2^64.

Complex tokens can be added, which makes very suitable for reading binary files or network messages:

 ExtendedHeader = {
    len: 10,

    get: (buf, off) => {
      return {
        // Extended header size
        size: Token.UINT32_BE.get(buf, off),
        // Extended Flags
        extendedFlags: Token.UINT16_BE.get(buf, off + 4),
        // Size of padding
        sizeOfPadding: Token.UINT32_BE.get(buf, off + 6),
        // CRC data present
        crcDataPresent: common.strtokBITSET.get(buf, off + 4, 31)
      };
    }
  };