strtok3

strtok3

A promise-based streaming tokenizer for Node.js and browsers.

The strtok3 module provides several methods for creating a tokenizer from various input sources. Designed for:

• Seamless support in streaming environments.
• Efficiently decode binary data, strings, and numbers.
• Reading predefined or custom tokens.
• Offering tokenizers for reading from files, streams or Uint8Arrays.

Features

strtok3 can read from:

• Files, using a file path as input.
• Node.js streams.
• WHATWG ReadableStream objects containing Uint8Array chunks.
• Buffer or Uint8Array.
• Blob or File objects.
• HTTP chunked transfer provided by @tokenizer/http.
• Amazon S3 chunks with @tokenizer/s3.

Installation

npm install strtok3

Compatibility

Starting with version 7, the module has migrated from CommonJS to pure ECMAScript Module (ESM). The distributed JavaScript codebase is compliant with the ECMAScript 2020 (11th Edition) standard.

Requires a modern browser, Node.js (V8) ≥ 18 engine or Bun (JavaScriptCore) ≥ 1.2. It can also be used in a browser environment when bundled with a module bundler.

For TypeScript CommonJS backward compatibility, you can use load-esm.

Use Node.js 24 LTS to build the project and install development dependencies. This build requirement is separate from the Node.js ≥ 18 runtime requirement in package.json. CI builds once with Node.js 24, then runs the compiled JavaScript unit tests across the Node.js compatibility matrix and Bun 1.2.x, 1.3.x, and 1.4.x. Coverage is collected in a separate Node.js 24 job.

For local development, running the TypeScript tests with tsx requires Node.js 18.19+ (18.x) or Node.js ≥ 20.6.0.

Support the Project

If you find this project useful and would like to support its development, consider sponsoring or contributing:

• Become a sponsor to Borewit

• Buy me a coffee:

  

API Documentation

strtok3 methods

Use one of the methods to instantiate an abstract tokenizer:

• fromBlob
• fromBuffer
• fromFile*
• fromStream*
• fromWebStream

[!NOTE] fromFile requires Node.js file-system APIs, and fromStream requires a Node.js readable stream. In browsers, use fromBlob, fromBuffer, or fromWebStream.

All methods return a Tokenizer, either directly or via a promise.

fromBlob() function

Create a tokenizer from a Blob.

function fromBlob(blob: Blob, options?: ITokenizerOptions): BlobTokenizer

Parameter	Optional	Type	Description
blob	no	Blob	Blob or File to read from
options	yes	ITokenizerOptions	Tokenizer options

Returns a tokenizer.

import { fromBlob } from 'strtok3';
import * as Token from 'token-types';

async function parse() {
  const blob = new Blob([new Uint8Array([42])]);

  const tokenizer = fromBlob(blob);

  const myUint8Number = await tokenizer.readToken(Token.UINT8);
  console.log(`My number: ${myUint8Number}`);   
}

parse();

fromBuffer() function

Create a tokenizer from memory (Uint8Array or Node.js Buffer).

function fromBuffer(uint8Array: Uint8Array, options?: ITokenizerOptions): BufferTokenizer

Parameter	Optional	Type	Description
uint8Array	no	Uint8Array	Buffer or Uint8Array to read from
options	yes	ITokenizerOptions	Tokenizer options

Returns a tokenizer.

import { fromBuffer } from 'strtok3';
import * as Token from 'token-types';

const tokenizer = fromBuffer(new Uint8Array([42]));

async function parse() {
  const myUint8Number = await tokenizer.readToken(Token.UINT8);
  console.log(`My number: ${myUint8Number}`);
}

parse();

fromFile function

Creates a tokenizer from a local file.

function fromFile(sourceFilePath: string, options?: ITokenizerOptions): Promise<FileTokenizer>

Parameter	Type	Description
sourceFilePath	string	Path to file to read from
options	ITokenizerOptions	Optional tokenizer options

[!NOTE]

• Only available for Node.js engines
• fromFile automatically embeds file information

A Promise resolving to a tokenizer which can be used to parse a file.

import { fromFile } from 'strtok3';
import * as Token from 'token-types';

async function parse() {
  const tokenizer = await fromFile('somefile.bin');
  try {
    const myNumber = await tokenizer.readToken(Token.UINT8);
    console.log(`My number: ${myNumber}`);
  } finally {
    await tokenizer.close(); // Close the file
  }
}

parse();

fromStream() function

Create a tokenizer from a Node.js readable stream containing binary data.

function fromStream(stream: Readable, options?: ITokenizerOptions): Promise<ReadStreamTokenizer>

Parameter	Optional	Type	Description
stream	no	Readable	Node.js readable stream to read from
options	yes	ITokenizerOptions	Tokenizer options

When importing from strtok3 in Node.js, returns a promise resolving to a tokenizer. For streams created by createReadStream, the file path and size are added to tokenizer.fileInfo. The strtok3/core entry point also exports fromStream, but returns the tokenizer synchronously without looking up file information.

import { createReadStream } from 'node:fs';
import { fromStream } from 'strtok3';
import * as Token from 'token-types';

async function parse() {
  const stream = createReadStream('somefile.bin');
  try {
    const tokenizer = await fromStream(stream);
    try {
      const myNumber = await tokenizer.readToken(Token.UINT8);
      console.log(`My number: ${myNumber}`);
    } finally {
      await tokenizer.close();
    }
  } finally {
    stream.destroy();
  }
}

parse();

fromWebStream() function

Create a tokenizer from a WHATWG ReadableStream containing Uint8Array chunks.

Uses a BYOB reader when supported by the stream, otherwise a default reader. Source cleanup differs between these readers; see close() and abort().

function fromWebStream(webStream: AnyWebByteStream, options?: ITokenizerOptions): ReadStreamTokenizer

Parameter	Optional	Type	Description
webStream	no	ReadableStream	WHATWG ReadableStream to read from
options	yes	ITokenizerOptions	Tokenizer options

Returns a tokenizer.

import { fromWebStream } from 'strtok3';
import * as Token from 'token-types';

async function parse() {
  const tokenizer = fromWebStream(readableStream);
  try {
    const myUint8Number = await tokenizer.readToken(Token.UINT8);
    console.log(`My number: ${myUint8Number}`);
  } finally {
    await tokenizer.close();
  }
}

parse();

Tokenizer object

The tokenizer is an abstraction of a stream, file or Uint8Array, allowing reading or peeking from the stream. It can also be translated in chunked reads, as done in @tokenizer/http;

Key Features:

• Supports seeking within the stream using tokenizer.ignore().
• Offers peek methods to preview data without advancing the read pointer.
• Maintains the read position via tokenizer.position.
• Supports random access for buffer, blob, and file tokenizers.

Tokenizer functions

Read methods advance the stream pointer, while peek methods do not.

There are two kinds of functions:

• read methods: used to read a token or bytes into a buffer from the tokenizer. The position of the tokenizer-stream will advance with the number of bytes read.
• peek methods: same as the read, but it will not advance the pointer. It allows to read (peek) ahead.

readBuffer function

Read data from the tokenizer into provided "buffer" (Uint8Array). readBuffer(buffer, options?)

readBuffer(buffer: Uint8Array, options?: IReadChunkOptions): Promise<number>;

Parameter	Type	Description
buffer	Buffer | Uint8Array	Target buffer to write the data read to
options	IReadChunkOptions	Optional settings for the length, position, and handling of partial reads

Return promise with number of bytes read. The number of bytes read may be less than requested if the mayBeLess flag is set.

peekBuffer function

Peek (read ahead), from tokenizer, into the buffer without advancing the stream pointer.

peekBuffer(uint8Array: Uint8Array, options?: IReadChunkOptions): Promise<number>;

Parameter	Type	Description
buffer	Buffer | Uint8Array	Target buffer to write the data read (peeked) to.
options	IReadChunkOptions	Optional settings for the length, position, and handling of partial reads

Return value Promise<number> Promise with number of bytes read. The number of bytes read may be less if the mayBeLess flag was set.

readToken function

Read a token from the tokenizer-stream.

readToken<Value>(token: IGetToken<Value>, position: number = this.position): Promise<Value>

Parameter	Type	Description
token	IGetToken	Token to read from the tokenizer-stream.
position?	number	Offset where to begin reading. Defaults to tokenizer.position when omitted.

Returns a Promise<Value> resolving to the decoded token value. Rejects with EndOfStreamError if fewer than token.len bytes are available.

peekToken function

Peek a token from the tokenizer.

peekToken<Value>(token: IGetToken<Value>, position: number | null = this.position): Promise<Value>

Parameter	Type	Description
token	IGetToken	Token to peek from the tokenizer-stream.
position?	number | null	Offset where to begin peeking. Defaults to tokenizer.position when omitted or null.

Return a promise with the token value peeked from the tokenizer.

readNumber function

Read a numeric token from the tokenizer.

readNumber(token: IToken<number>): Promise<number>

Parameter	Type	Description
token	IToken	Numeric token to read from the tokenizer-stream.

A promise resolving to a numeric value read and decoded from the tokenizer-stream.

peekNumber function

Peek a numeric token without advancing tokenizer.position.

peekNumber(token: IToken<number>): Promise<number>

Returns a promise resolving to the decoded numeric value. Like readNumber, rejects with EndOfStreamError if the token cannot be read in full.

ignore function

Advance the offset pointer with the token number of bytes provided.

ignore(length: number): Promise<number>

Parameter	Type	Description
length	number	Non-negative number of bytes to ignore. Advances tokenizer.position.

Returns a promise resolving to the number of bytes ignored. Buffer, blob, and file tokenizers stop at the known file size. Stream tokenizers reject with EndOfStreamError if the requested bytes are unavailable.

close function

Clean up resources, such as closing a file pointer if applicable.

close(): Promise<void>

Await this method when finished with a tokenizer. For Node.js streams, it does not destroy the underlying stream; the caller remains responsible for its lifecycle.

For Web streams, close() releases the reader lock. A BYOB reader leaves the source uncancelled, allowing another reader to continue reading it. A default reader cancels the source before releasing the lock, discarding remaining data. Bytes already read into tokenizer buffers, including peeked bytes, are not returned to the source.

abort function

Abort pending asynchronous stream operations.

abort(): Promise<void>

This is a no-op for buffer, blob, and file tokenizers.

For Web streams, abort() releases a BYOB reader's lock without cancelling the source. A default reader cancels the source but keeps its lock until close() is called. An AbortSignal invokes the same abort behavior.

supportsRandomAccess function

supportsRandomAccess(): boolean

Returns true for buffer, blob, and file tokenizers, and false for stream tokenizers. When random access is supported, read and peek methods can use positions before tokenizer.position.

setPosition function

Available on buffer, blob, and file tokenizers (IRandomAccessTokenizer).

setPosition(position: number): void

Sets the current byte offset without reading data.

Tokenizer attributes

• fileInfo

  Object containing optional file information, see IFileInfo.

• position

  Pointer to the current position in the tokenizer stream. For stream tokenizers, a position provided to a read or peek method must be at least this value. Buffer, blob, and file tokenizers also support earlier positions.

ITokenizerOptions interface

Set these options once when creating a tokenizer. The configured limit applies to subsequent reads and peeks for that tokenizer; no per-read limit is needed. Each attribute is optional:

Attribute	Type	Description
maxBufferSize	number	Maximum bytes in a dynamically sized tokenizer buffer or stream lookahead span. Disabled by default (Infinity). Must be a positive safe integer or Infinity.
fileInfo	IFileInfo	Additional metadata about the input
onClose	() => Promise<void>	Asynchronous callback invoked when a buffer, blob, or file tokenizer is closed
abortSignal	AbortSignal	Signal used to abort asynchronous stream operations

Creating a tokenizer with an already-aborted abortSignal throws or rejects with AbortError before acquiring file handles or stream readers. Signal listeners are removed when the signal fires or the tokenizer is closed, so closed tokenizers can be collected even when a signal is reused.

When maxBufferSize is configured, readToken() and peekToken() reject oversized buffers with BufferSizeError before allocating or consuming input. The error exposes requestedSize and maxBufferSize. Token lengths and positions must be non-negative safe integers. When the input size is known, a token extending beyond the input rejects with EndOfStreamError before allocation.

For streams, the lookahead limit includes bytes between the current position and the requested peek position. mayBeLess: true does not bypass this limit. With a finite cap, reads into caller-provided buffers, Blob reads and ignore() use bounded chunks where internal buffers are needed, so their total requested length may exceed maxBufferSize.

This limit does not cap total process memory, caller-owned input/output buffers, or allocations inside a token's get() method. Stream sources may also supply chunks larger than this limit. The cap is opt-in to preserve existing buffer sizes. Applications parsing untrusted input should set a finite limit, for example 64 MiB. Omit the option or set it to Infinity to disable the cap.

import { BufferSizeError, fromFile } from 'strtok3';

const tokenizer = await fromFile('somefile.bin', { maxBufferSize: 64 * 1024 * 1024 });
try {
  // Read or peek tokens here.
} catch (error) {
  if (error instanceof BufferSizeError) {
    console.error(`Requested ${error.requestedSize} bytes; limit is ${error.maxBufferSize}`);
  } else {
    throw error;
  }
} finally {
  await tokenizer.close();
}

IReadChunkOptions interface

Each attribute is optional:

Attribute	Type	Description
length	number	Requested number of bytes to read or peek. Defaults to the target buffer length.
position	number	Byte offset to read or peek from. Defaults to tokenizer.position when omitted. Earlier positions require random access.
mayBeLess	boolean	If true, allows fewer than the requested number of bytes to be read or peeked instead of throwing EndOfStreamError. Defaults to false.

Example usage:

  const bytesPeeked = await tokenizer.peekBuffer(buffer, {mayBeLess: true});

IFileInfo interface

Provides optional metadata about the file being tokenized.

Attribute	Type	Description
size	number	File size in bytes
mimeType	string	MIME-type of file.
path	string	File path
url	string	File URL

Token object

The token is basically a description of what to read from the tokenizer-stream. A basic set of token types can be found here: token-types.

A token is something which implements the following interface:

export interface IGetToken<T> {

  /**
   * Length in bytes of encoded value
   */
  len: number;

  /**
   * Decode value from buffer at offset
   * @param buf Buffer to read the decoded value from
   * @param off Decode offset
   */
  get(buf: Uint8Array, off: number): T;
}

The tokenizer reads token.len bytes from the tokenizer-stream into a Buffer. The token.get will be called with the Buffer. token.get is responsible for conversion from the buffer to the desired output type.

Working with a fetch response

Pass the response's WHATWG readable stream directly to fromWebStream.

import { fromWebStream } from 'strtok3';
import * as Token from 'token-types';

(async () => {

  const response = await fetch(url);
  if (!response.ok || !response.body) {
    throw new Error(`Cannot read response: ${response.status}`);
  }

  const tokenizer = fromWebStream(response.body);
  try {
    const myNumber = await tokenizer.readToken(Token.UINT8);
    console.log(`My number: ${myNumber}`);
  } finally {
    await tokenizer.close();
  }
})();

Dependencies

Dependencies:

• @tokenizer/token: Provides token definitions and utilities used by strtok3 for interpreting binary data.

Contributing

See the contribution guide for code style and development commands.

Licence

This project is licensed under the MIT License. Feel free to use, modify, and distribute as needed.

Similar packages

Other packages similar to strtok3

binary-parser

binary-parser is a package for building efficient binary data parsers. It is similar to strtok3 in that it helps parse binary data, but it uses a declarative approach to define the structure of the binary data instead of the imperative approach used by strtok3.