token-types

What is token-types?

The token-types npm package provides a collection of token parsers for binary data types, facilitating the reading and interpretation of various data types from binary streams. This is particularly useful in applications dealing with multimedia files, where metadata and content need to be extracted or interpreted from binary formats.

What are token-types's main functionalities?

Reading integer values

This feature allows the reading of unsigned 32-bit integers from a buffer in little-endian format. It is useful for parsing low-level binary data structures.

const Token = require('token-types');

const uint32Token = Token.UINT32_LE;
const buffer = Buffer.from([0x01, 0x00, 0x00, 0x00]);
const value = uint32Token.get(buffer, 0);
console.log(value); // Outputs: 1

Reading string values

This feature enables the extraction of ASCII strings from binary data. It supports specifying the length of the string and the encoding, making it versatile for reading text data embedded in binary streams.

const Token = require('token-types');

const stringToken = new Token.StringType(5, 'ascii');
const buffer = Buffer.from('hello');
const value = stringToken.get(buffer, 0);
console.log(value); // Outputs: 'hello'

Other packages similar to token-types

binary-parser

binary-parser offers functionality to build a schema for parsing binary data, which is somewhat similar to token-types. However, binary-parser is more comprehensive in defining complex parsing schemas, making it suitable for applications that require detailed and structured binary data interpretation.

README

token-types

A primitive token library used to read and write from a Uint8Array. Although it is possible to use this module directly, it is primarily designed to be used with strtok3 tokenizer.

Compatibility

Module: version 5 migrated from CommonJS to pure ECMAScript Module (ESM). JavaScript is compliant with ECMAScript 2019 (ES10).

Installation

npm install --save token-types

Usually in combination with strtok3:

npm install --save strtok3

Using TypeScript you should also install @tokenizer/token as a development dependency:

npm install --save-dev @tokenizer/token

Example

import * as strtok3 from 'strtok3';
import * as token from 'token-types';

(async () => {

  const tokenizer = await strtok3.fromFile("somefile.bin");
  try {
    const myNumber = await tokenizer.readToken(token.Float32_BE);
    console.log(`My number: ${myNumber}`);
  } finally {
    tokenizer.close(); // Close the file
  }
})();

Tokens

Numeric tokens

token-types supports a wide variety of numeric tokens out of the box:

Token	Number	Bits	Endianness
UINT8	Unsigned integer	8	n/a
UINT16_BE	Unsigned integer	16	big endian
UINT16_LE	Unsigned integer	16	little endian
UINT24_BE	Unsigned integer	24	big endian
UINT24_LE	Unsigned integer	24	little endian
UINT32_BE	Unsigned integer	32	big endian
UINT32_LE	Unsigned integer	32	little endian
UINT64_BE	Unsigned integer	64	big endian
UINT64_LE*	Unsigned integer	64	little endian
INT8	Signed integer	8	n/a
INT16_BE	Signed integer	16	big endian
INT16_LE	Signed integer	16	little endian
INT24_BE	Signed integer	24	big endian
INT24_LE	Signed integer	24	little endian
INT32_BE	Signed integer	32	big endian
INT32_LE	Signed integer	32	little endian
INT64_BE	Signed integer	64	big endian
INT64_LE*	Signed integer	64	little endian
Float16_BE	IEEE 754 float	16	big endian
Float16_LE	IEEE 754 float	16	little endian
Float32_BE	IEEE 754 float	32	big endian
Float32_LE	IEEE 754 float	32	little endian
Float64_BE	IEEE 754 float	64	big endian
Float64_LE	IEEE 754 float	64	little endian
Float80_BE*	IEEE 754 float	80	big endian
Float80_LE*	IEEE 754 float	80	little endian

(*) The tokens exceed the JavaScript IEEE 754 64-bit Floating Point precision, decoding and encoding is best effort based.

String tokens

StringType decoding is implemented using TextDecoder which supports a large number of encodings including but not limited to:

• UTF-8 (the default)
• Windows-1252
• ISO-8859-1

Check out the MDN web docs for the TextDecoder for a complete list

Custom tokens

Custom tokens can be added, suitable for reading binary files or network messages:

  ExtendedHeader = {
    len: 10,

    get: (buf, off) => {
      return {
        // Extended header size
        size: Token.UINT32_BE.get(buf, off),
        // Extended Flags
        extendedFlags: Token.UINT16_BE.get(buf, off + 4),
        // Size of padding
        sizeOfPadding: Token.UINT32_BE.get(buf, off + 6),
        // CRC data present
        crcDataPresent: common.strtokBITSET.get(buf, off + 4, 31)
      };
    }
  };