Security News
ESLint is Now Language-Agnostic: Linting JSON, Markdown, and Beyond
ESLint has added JSON and Markdown linting support with new officially-supported plugins, expanding its versatility beyond JavaScript.
The strtok3 npm package is a streaming tokenizer for Buffer and string inputs in Node.js. It allows developers to parse through binary data or strings efficiently by defining tokenizers that can extract pieces of data sequentially. This is particularly useful for reading binary files or network streams where data structures are defined in terms of sequences or patterns of bytes.
Tokenizing fixed-length binary data
This feature allows for reading fixed-length binary data from a file. The example demonstrates reading a 32-bit unsigned integer from the beginning of a file.
const strtok3 = require('strtok3');
const { token } = require('strtok3/core');
async function parseBinaryFile(filePath) {
const tokenizer = await strtok3.fromFile(filePath);
const header = await tokenizer.readToken(token.UINT32_LE);
console.log('Header:', header);
}
Tokenizing a stream of data
This feature is used for tokenizing data directly from a stream. The example shows how to read a size as a 32-bit unsigned integer and then read a buffer of that size from the stream.
const strtok3 = require('strtok3');
const { token } = require('strtok3/core');
const fs = require('fs');
const stream = fs.createReadStream('path/to/file');
const tokenizer = strtok3.fromStream(stream);
async function readData() {
const size = await tokenizer.readToken(token.UINT32_LE);
const data = await tokenizer.readToken(new token.BufferType(size));
console.log('Data:', data.toString());
}
binary-parser is a package for building efficient binary data parsers. It is similar to strtok3 in that it helps parse binary data, but it uses a declarative approach to define the structure of the binary data instead of the imperative approach used by strtok3.
A promise based streaming tokenizer for Node.js and browsers. This node module is a successor of strtok2.
The strtok3
contains a few methods to turn different input into a tokenizer. Designed to
It can read from:
npm install --save strtok3
Use one of the methods to instantiate an abstract tokenizer:
All of the strtok3 methods return a tokenizer, either directly or via a promise.
strtok3.fromFile()
Parameter | Type | Description |
---|---|---|
path | Path to file (string) | Path to file to read from |
Returns, via a promise, a tokenizer which can be used to parse a file.
const strtok3 = require('strtok3');
const Token = require('token-types');
(async () => {
const tokenizer = await strtok3.fromFile("somefile.bin");
try {
const myNumber = await tokenizer.readToken(Token.UINT8);
console.log(`My number: ${myNumber}`);
} finally {
tokenizer.close(); // Close the file
}
})();
strtok3.fromStream()
Create tokenizer from a node.js readable stream.
Parameter | Type | Description |
---|---|---|
stream | Readable | Stream to read from |
Returns a tokenizer, via a Promise, which can be used to parse a buffer.
const strtok3 = require('strtok3');
const Token = require('token-types');
strtok3.fromStream(stream).then(tokenizer => {
return tokenizer.readToken(Token.UINT8).then(myUint8Number => {
console.log(`My number: ${myUint8Number}`);
});
});
strtok3.fromBuffer()
Parameter | Type | Description |
---|---|---|
buffer | Buffer | Buffer to read from |
Returns a tokenizer which can be used to parse the provided buffer.
const strtok3 = require('strtok3');
const tokenizer = strtok3.fromBuffer(buffer);
tokenizer.readToken(Token.UINT8).then(myUint8Number => {
console.log(`My number: ${myUint8Number}`);
});
The tokenizer allows us to read or peek from the tokenizer-stream. The tokenizer-stream is an abstraction of a stream, file or Buffer. It can also be translated in chunked reads, as done in @tokenizer/http;
What is the difference with Nodejs.js stream?
tokenizer.ignore()
The tokenizer.position keeps tracks of
tokenizer.fileSize
Optional attribute of the total file or stream length in bytes
tokenizer.position
Pointer to the current position in the tokenizer stream. If a position is provided to a read or peek method, is should be, at least, equal or greater than this value.
There are to groups of methods
tokenizer.readBuffer()
Read buffer from stream.
readBuffer(buffer, offset?, length?, position?, maybeless?)
Parameter | Type | Description |
---|---|---|
buffer | Buffer | Uint8Array | Target buffer to write the data read to |
offset? | number | The offset in the buffer to start writing at; if not provided, start at 0 |
length? | number | An integer specifying the number of bytes to read |
position? | number | An integer specifying where to begin reading from in the file. If position is null, data will be read from the current file position. |
maybeless? | boolean | If set, will not throw an EOF error if the less then the requested length could be read |
Return value Promise<number>
Promise with number of bytes read
tokenizer.peekBuffer()
Peek (read ahead) buffer from tokenizer
peekBuffer(buffer, offset?, length?, position?, maybeless?)
Parameter | Type | Description |
---|---|---|
buffer | Buffer | Uint8Array | Target buffer to write the data read (peeked) to |
offset? | number | The offset in the buffer to start writing at; if not provided, start at 0 |
length? | number | The number of bytes to read. |
position? | number | Offset where to begin reading within the file. If position is null, data will be read from the current file position. |
maybeless? | boolean | If set, will not throw an EOF error if the less then the requested length could be read |
Return value Promise<number>
Promise with number of bytes read
tokenizer.readToken()
Read a token from the tokenizer-stream.
readToken(token, position?, maybeless?)
Parameter | Type | Description |
---|---|---|
token | IGetToken | Token to read from the tokenizer-stream. |
position? | number | Offset where to begin reading within the file. If position is null, data will be read from the current file position. |
maybeless? | boolean | If set, will not throw an EOF error if the less then the requested length could be read. |
Return value Promise<T>
. Promise with token value read from the tokenizer-stream.
tokenizer.peekToken()
Peek a token from the tokenizer-stream.
peekToken(token, position?, maybeless?)
Parameter | Type | Description |
---|---|---|
token | IGetToken | Token to read from the tokenizer-stream. |
position? | number | Offset where to begin reading within the file. If position is null, data will be read from the current file position. |
maybeless? | boolean | If set, will not throw an EOF error if the less then the requested length could be read |
Return value Promise<T>
Promise with token value peeked from the tokenizer-stream.
tokenizer.readNumber()
Peek a numeric token from the tokenizer-stream.
readNumber(token)
Parameter | Type | Description |
---|---|---|
token | IGetToken | Numeric token to read from the tokenizer-stream. |
Return value Promise<number>
Promise with number peeked from the tokenizer-stream.
tokenizer.ignore()
Peek a numeric token from the tokenizer-stream.
ignore(length)
Parameter | Type | Description |
---|---|---|
ignore | number | Numeric of bytes to ignore. Will advance the tokenizer.position |
Return value Promise<number>
Promise with number peeked from the tokenizer-stream.
tokenizer.close()
Clean up resources, such as closing a file pointer if applicable.
The token is basically a description what to read form the tokenizer-stream. A basic set of token types can be found here: token-types.
A token is something which implements the following interface:
export interface IGetToken<T> {
/**
* Length in bytes of encoded value
*/
len: number;
/**
* Decode value from buffer at offset
* @param buf Buffer to read the decoded value from
* @param off Decode offset
*/
get(buf: Buffer, off: number): T;
}
The tokenizer reads token.len
bytes from the tokenizer-stream into a Buffer.
The token.get
will be called with the Buffer. token.get
is responsible for conversion from the buffer to the desired output type.
To exclude fs based dependencies, you can use a submodule-import from 'strtok3/lib/core'.
function | 'strtok3' | 'strtok3/lib/core' |
---|---|---|
parseBuffer | ✓ | ✓ |
parseStream | ✓ | ✓ |
fromFile | ✓ |
To convert a Web-API readable stream into a Node.js readable stream, you can use readable-web-to-node-stream to convert one in another.
Example submodule-import:
const strtok3core = require('strtok3/lib/core'); // Submodule-import to prevent Node.js specific dependencies
const {ReadableWebToNodeStream} = require('readable-web-to-node-stream');
(async () => {
const response = await fetch(url);
const readableWebStream = response.body; // Web-API readable stream
const nodeStream = new ReadableWebToNodeStream(readableWebStream); // convert to Node.js readable stream
const tokenizer = strtok3core.fromStream(nodeStream); // And we now have tokenizer in a web environment
})();
FAQs
A promise based streaming tokenizer
The npm package strtok3 receives a total of 4,396,688 weekly downloads. As such, strtok3 popularity was classified as popular.
We found that strtok3 demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
ESLint has added JSON and Markdown linting support with new officially-supported plugins, expanding its versatility beyond JavaScript.
Security News
Members Hub is conducting large-scale campaigns to artificially boost Discord server metrics, undermining community trust and platform integrity.
Security News
NIST has failed to meet its self-imposed deadline of clearing the NVD's backlog by the end of the fiscal year. Meanwhile, CVE's awaiting analysis have increased by 33% since June.