Security News
ESLint is Now Language-Agnostic: Linting JSON, Markdown, and Beyond
ESLint has added JSON and Markdown linting support with new officially-supported plugins, expanding its versatility beyond JavaScript.
The strtok3 npm package is a streaming tokenizer for Buffer and string inputs in Node.js. It allows developers to parse through binary data or strings efficiently by defining tokenizers that can extract pieces of data sequentially. This is particularly useful for reading binary files or network streams where data structures are defined in terms of sequences or patterns of bytes.
Tokenizing fixed-length binary data
This feature allows for reading fixed-length binary data from a file. The example demonstrates reading a 32-bit unsigned integer from the beginning of a file.
const strtok3 = require('strtok3');
const { token } = require('strtok3/core');
async function parseBinaryFile(filePath) {
const tokenizer = await strtok3.fromFile(filePath);
const header = await tokenizer.readToken(token.UINT32_LE);
console.log('Header:', header);
}
Tokenizing a stream of data
This feature is used for tokenizing data directly from a stream. The example shows how to read a size as a 32-bit unsigned integer and then read a buffer of that size from the stream.
const strtok3 = require('strtok3');
const { token } = require('strtok3/core');
const fs = require('fs');
const stream = fs.createReadStream('path/to/file');
const tokenizer = strtok3.fromStream(stream);
async function readData() {
const size = await tokenizer.readToken(token.UINT32_LE);
const data = await tokenizer.readToken(new token.BufferType(size));
console.log('Data:', data.toString());
}
binary-parser is a package for building efficient binary data parsers. It is similar to strtok3 in that it helps parse binary data, but it uses a declarative approach to define the structure of the binary data instead of the imperative approach used by strtok3.
A promise based streaming tokenizer for Node.js and browsers. This node module is a successor of strtok2.
The strtok3
contains a few methods to turn different input into a tokenizer. Designed to
It can read from:
Use one of the methods to instantiate an abstract tokenizer:
All of the strtok3 methods return a tokenizer, either directly or via a promise.
strtok3.fromFile()
Returns, via a promise, a tokenizer which can be used to parse a file.
import strtok3 from 'strtok3';
import Token from 'token-types';
strtok3.fromFile("somefile.bin").then(tokenizer => {
return tokenizer.readToken<number>(Token.UINT8).then(myUint8Number => {
console.log("My number: %s", myUint8Number);
});
})
strtok3.fromStream()
Create tokenizer from a node.js readable stream.
Parameter | Type | Description |
---|---|---|
stream | Readable | Stream to read from |
import strtok3 from 'strtok3';
import Token from 'token-types';
strtok3.fromStream(stream).then(tokenizer => {
return tokenizer.readToken<number>(Token.UINT8).then(myUint8Number => {
console.log("My number: %s", myUint8Number);
});
});
Returns a tokenizer, via a Promise, which can be used to parse a buffer.
strtok3.fromBuffer()
Returns a tokenizer which can be used to parse a buffer.
import strtok3 from 'strtok3';
const tokenizer = strtok3.fromBuffer(buffer);
tokenizer.readToken<number>(Token.UINT8).then(myUint8Number => {
console.log("My number: %s", myUint8Number);
});
The tokenizer allows us to read or peek from the tokenizer-stream. The tokenizer-stream is an abstraction of a stream, file or Buffer. It can also be translated in chunked reads, as done in streaming-http-token-reader;
What is the difference with Nodejs.js stream?
tokenizer.ignore()
The tokenizer.position keeps tracks of
tokenizer.fileSize
Optional attribute of the total file or stream length in bytes
tokenizer.position
Pointer to the current position in the tokenizer stream. If a position is provided to a read or peek method, is should be, at least, equal or greater than this value.
There are to groups of methods
tokenizer.readBuffer()
Read buffer from stream.
readBuffer(buffer, offset?, length?, position?, maybeless?)
Parameter | Type | Description |
---|---|---|
buffer | Buffer | Uint8Array | Target buffer to write the data read to |
offset? | number | The offset in the buffer to start writing at; if not provided, start at 0 |
length? | number | An integer specifying the number of bytes to read |
position? | number | An integer specifying where to begin reading from in the file. If position is null, data will be read from the current file position. |
maybeless? | boolean | If set, will not throw an EOF error if the less then the requested length could be read |
Return value Promise<number>
Promise with number of bytes read
tokenizer.peekBuffer()
Peek (read ahead) buffer from tokenizer
peekBuffer(buffer, offset?, length?, position?, maybeless?)
Parameter | Type | Description |
---|---|---|
buffer | Buffer | Uint8Array | Target buffer to write the data read (peeked) to |
offset? | number | The offset in the buffer to start writing at; if not provided, start at 0 |
length? | number | The number of bytes to read. |
position? | number | Offset where to begin reading within the file. If position is null, data will be read from the current file position. |
maybeless? | boolean | If set, will not throw an EOF error if the less then the requested length could be read |
<<<<<<< HEAD |
Return value Promise<number>
Promise with number of bytes read
tokenizer.readToken()
Read a token from the tokenizer-stream.
readToken(token, position?, maybeless?)
Parameter | Type | Description |
---|---|---|
token | IGetToken | Token to read from the tokenizer-stream. |
position? | number | Offset where to begin reading within the file. If position is null, data will be read from the current file position. |
maybeless? | boolean | If set, will not throw an EOF error if the less then the requested length could be read. |
Return value Promise<T>
. Promise with token value read from the tokenizer-stream.
tokenizer.peekToken()
Peek a token from the tokenizer-stream.
peekToken(token, position?, maybeless?)
Parameter | Type | Description |
---|---|---|
token | IGetToken | Token to read from the tokenizer-stream. |
position? | number | Offset where to begin reading within the file. If position is null, data will be read from the current file position. |
maybeless? | boolean | If set, will not throw an EOF error if the less then the requested length could be read |
Return value Promise<T>
Promise with token value peeked from the tokenizer-stream.
tokenizer.readNumber()
Peek a numeric token from the tokenizer-stream.
readNumber(token)
Parameter | Type | Description |
---|---|---|
token | IGetToken | Numeric token to read from the tokenizer-stream. |
Return value Promise<number>
Promise with number peeked from the tokenizer-stream.
tokenizer.ignore()
Peek a numeric token from the tokenizer-stream.
ignore(length)
Parameter | Type | Description |
---|---|---|
ignore | number | Numeric of bytes to ignore. Will advance the tokenizer.position |
Return value Promise<number>
Promise with number peeked from the tokenizer-stream.
tokenizer.close()
Clean up resources, such as closing a file pointer if applicable.
=======
Return value Promise<number>
Promise with number of bytes read
tokenizer.readToken()
Read a token from the tokenizer-stream.
readToken(token, position?, maybeless?)
Parameter | Type | Description |
---|---|---|
token | IGetToken | Token to read from the tokenizer-stream. |
position? | number | Offset where to begin reading within the file. If position is null, data will be read from the current file position. |
maybeless? | boolean | If set, will not throw an EOF error if the less then the requested length could be read. |
Return value Promise<T>
. Promise with token value read from the tokenizer-stream.
tokenizer.peekToken()
Peek a token from the tokenizer-stream.
peekToken(token, position?, maybeless?)
Parameter | Type | Description |
---|---|---|
token | IGetToken | Token to read from the tokenizer-stream. |
position? | number | Offset where to begin reading within the file. If position is null, data will be read from the current file position. |
maybeless? | boolean | If set, will not throw an EOF error if the less then the requested length could be read |
Return value Promise<T>
Promise with token value peeked from the tokenizer-stream.
tokenizer.readNumber()
Peek a numeric token from the tokenizer-stream.
readNumber(token)
Parameter | Type | Description |
---|---|---|
token | IGetToken | Numeric token to read from the tokenizer-stream. |
Return value Promise<number>
Promise with number peeked from the tokenizer-stream.
tokenizer.ignore()
Peek a numeric token from the tokenizer-stream.
ignore(length)
Parameter | Type | Description |
---|---|---|
ignore | number | Numeric of bytes to ignore. Will advance the tokenizer.position |
Return value Promise<number>
Promise with number peeked from the tokenizer-stream.
tokenizer.close()
Clean up resources, such as closing a file pointer if applicable.
44aceaf19e8d3d1ef8b40548f7c78f7502f138fe
The token is basically a description what to read form the tokenizer-stream. A basic set of token types can be found here: token-types.
Below is an example of parsing the the first byte from a readable stream as an unsigned-integer:
import strtok3 from 'strtok3';
import Token from 'token-types';
let readableStream; // stream.Readable;
strtok3.fromStream(readableStream).then(tokenizer => {
return tokenizer.readToken<number>(Token.UINT8).then(myUint8Number => {
console.log("My number: %s", myUint8Number);
});
})
To exclude fs based dependencies, you can use a submodule-import from 'strtok3/lib/core'.
function | 'strtok3' | 'strtok3/lib/core' |
---|---|---|
parseBuffer | ✓ | ✓ |
parseStream | ✓ | ✓ |
fromFile | ✓ |
Example submodule-import:
import strtok3core from 'strtok3/lib/core';
const tokenizer = strtok3core.fromStream(stream);
If you plan to use fromStream
you need to polyfill:
FAQs
A promise based streaming tokenizer
The npm package strtok3 receives a total of 4,396,688 weekly downloads. As such, strtok3 popularity was classified as popular.
We found that strtok3 demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
ESLint has added JSON and Markdown linting support with new officially-supported plugins, expanding its versatility beyond JavaScript.
Security News
Members Hub is conducting large-scale campaigns to artificially boost Discord server metrics, undermining community trust and platform integrity.
Security News
NIST has failed to meet its self-imposed deadline of clearing the NVD's backlog by the end of the fiscal year. Meanwhile, CVE's awaiting analysis have increased by 33% since June.