strtok3

What is strtok3?

The strtok3 npm package is a streaming tokenizer for Buffer and string inputs in Node.js. It allows developers to parse through binary data or strings efficiently by defining tokenizers that can extract pieces of data sequentially. This is particularly useful for reading binary files or network streams where data structures are defined in terms of sequences or patterns of bytes.

What are strtok3's main functionalities?

Tokenizing fixed-length binary data

This feature allows for reading fixed-length binary data from a file. The example demonstrates reading a 32-bit unsigned integer from the beginning of a file.

const strtok3 = require('strtok3');
const { token } = require('strtok3/core');

async function parseBinaryFile(filePath) {
  const tokenizer = await strtok3.fromFile(filePath);
  const header = await tokenizer.readToken(token.UINT32_LE);
  console.log('Header:', header);
}

Tokenizing a stream of data

This feature is used for tokenizing data directly from a stream. The example shows how to read a size as a 32-bit unsigned integer and then read a buffer of that size from the stream.

const strtok3 = require('strtok3');
const { token } = require('strtok3/core');
const fs = require('fs');

const stream = fs.createReadStream('path/to/file');
const tokenizer = strtok3.fromStream(stream);

async function readData() {
  const size = await tokenizer.readToken(token.UINT32_LE);
  const data = await tokenizer.readToken(new token.BufferType(size));
  console.log('Data:', data.toString());
}

Other packages similar to strtok3

binary-parser

binary-parser is a package for building efficient binary data parsers. It is similar to strtok3 in that it helps parse binary data, but it uses a declarative approach to define the structure of the binary data instead of the imperative approach used by strtok3.

README

A promise based streaming tokenizer for NodeJS. This node module is a successor of strtok2.

Usage

The strtok3 contains one class, ReadStreamTokenizer, which is constructed with a a stream.Readable.

The ReadStreamTokenizer has one method readToken which takes a token as an argument and returns a Promise resolving the decoded token value.

The token is basically a description what to read form the stream. A basic set of token types can be found here: token-types.

Reading from a stream

Below is an example of parsing the the first byte from a readable stream as an unsigned-integer:

import {ReadStreamTokenizer} from "strtok3";
import * as stream from "stream";
import * as Token from "token-types";
    
let readableStream: stream.Readable;
// Assign readable

const streamTokenizer = new ReadStreamTokenizer(readableStream);

return streamTokenizer.readToken<number>(Token.UINT8).then((myUint8Number) => {
  console.log("My number: %s", myUint8Number);
});

Reading from a file

The same can be done from a file:

import * as fs from "fs-extra";
import {FileTokenizer} from "strtok3";
import * as Token from "token-types";
    
return fs.open("somefile.bin", "r").then((fd) => {
  const fileTokenizer = new FileTokenizer(fd);

  return fileTokenizer.readToken<number>(Token.UINT8).then((myUint8Number) => {
      console.log("My number: %s", myUint8Number);
    });
});