Binarcular
A library that allows you to read/write the contents of a binary file into/from a JSON Object,
taking care of all data type conversion using JavaScript-friendly values. You can search for data
by value, slice blocks into separate, meaningful structures or generate a binary downloadable
file, all inside your browser.
Practical use cases are:
- Validating whether a file header contains the appropriate description, automatically
converted to the data types
- Scanning a file for specific meta data to determine the location of other meaningful data
- Writing a binary file in your browser, without having to write meaningless byte values
repeatedly at ever increasing indices.
See API and Example below.
Compatibility
binarcular should work fine on Internet Explorer 10 and up. You can verify
support at runtime by querying the result of the isSupported()-method:
import { isSupported } from 'binarcular';
if ( isSupported( optRequire64bitConversion = false ) ) {
...do stuff!
} else {
...do other, less cool stuff! Actually, not cool at all!! >=/
}
NOTE: if you require support for 64-bit types there are additional requirements. Pass boolean true for optional argument optRequire64bitConversion to determine whether the environment supports 64-bit conversion.
Installation
You can get it via NPM:
npm install binarcular
Project integration
The parser is compatible with CommonJS / ES6 modules or can be included in a document
using AMD/RequireJS. See the contents of the /dist/ folder and include as your project sees fit.
API
The module exports the following:
import {
isSupported: fn( optRequire64bitConversion = false ),
types: Object<String>,
parse: async fn( dataSource, structureDefinition, optReadOffset = 0 ),
seek: async fn( uint8Array, searchStringOrByteArray, optReadOffset = 0 ),
write: async fn( uint8Array, structureDefinition, dataToWrite, optWriteOffset = 0 )
fileToByteArray: async fn( file, optSliceOffset = 0, optSliceSize = file.size )
byteArrayToFile: fn( uint8Array, filename, optMimeType = 'application/octet-stream' )
} from 'binarcular';
We'll look into each of these below:
Reading a chunk of data into an Object of a specific structure type
The parse method is reponsible for this:
async parse( dataSource, structureDefinition, optReadOffset = 0 )
Where:
- dataSource is the file to parse (can be either File, Blob, Uint8Array or (base64 encoded) String)
- structureDefinition is an Object defining a data structure (as described here)
- optReadOffset is a numerical index describing where in the file's ByteArray reading should start
this defaults to 0 to start at the beginning of the file.
When the Promise resolves, the result is the following structure:
{
data: Object,
end: Number,
error: Boolean,
byteArray: Uint8Array
}
If all has been read successfully, data is an Object that follows
the structure of wavHeader and is populated with the actual file data.
end describes at what offset in given file the structure's definition has ended.
This can be used for subsequent read operations where different data types are
extracted from the binary data.
If error is true, this indicates that something went wrong during parsing. The data
Object will be populated with all data that could've been harvested up until the error occurred.
This allow you to harvest what you can from corrupted files.
A note on using Uint8Array as dataSource
You can see that another property is defined in the result, namely byteArray.
If the dataSource provided to the parse method was a Uint8Array which you intend to
reuse inside your project, be sure to reassign your byteArray reference to the
returned instance.
The rationale here is that for minimal overhead, the ownership of the ByteArray's binary
content is transferred during the read operations. You will not be able to perform
any actions on your ByteArray without updating the reference.
Looking for a specific entry in a file
If you are working with a file where the content of interest is preceded by some
metadata at an arbitrary point, it makes sense to first look for this metadata
declaration so you know from where you can retrieve the actual data of interest.
For this purpose you can use seek:
async seek( uint8Array, searchStringOrByteArray, optReadOffset = 0 )
where:
- uint8Array is the ByteArray containing the binary data.
- searchStringOrByteArray can be either a String (in case the meta data is a
character sequence) or a Uint8Array holding a byte sequence that functions as the "search query".
- optReadOffset determines the offset within the data from where to start searching, this
defaults to 0 to read from the start.
The method returns a numerical index at which the data was found or Infinity
if no match were found.
Writing JSON as binary content
If you have a JSON structure that you wish to write into a binary file, you can do
so using write:
async write( uint8Array, structureDefinition, dataToWrite, optWriteOffset = 0 )
where:
- uint8Array is the ByteArray containing the binary data.
- structureDefinition is an Object defining a data structure (as described here)
- dataToWrite is an Object following the data structure, except the value here is the
data you wish to write in the binary file.
- optWriteOffset is the index at which data will be written. This defaults to 0 to
start writing at the beginning of the file. Data will be written for the length of
given structureDefinition, all existing data beyond this point will remain unchanged.
The result of this operation is the following:
{
data: Object,
end: Number,
error: Boolean,
byteArray: Uint8Array
}
Where byteArray should replace the reference of the byteArray you passed into
the method. This ByteArray contains the original data except that the data block starting
at requested optWriteOffset has been replaced with the binary equivalent of the dataToWrite-Object.
All data beyond the size of the written data block remains unchanged.
Converting a File reference to a ByteArray
async fileToByteArray( fileReference, optSliceOffset = 0, optSliceSize = fileReference.size )
where:
- fileReference is the File (or Blob) of which the contents should be read into a Uint8Array.
- optSliceOffset is the optional offset from where to read the data, defaults to 0 to
start from the beginning.
- optSliceSize is the optional size of the resulting ByteArray. This defaults to the
size of the file to read the file in its entirety. When using a custom optSliceOffset
overflow checking is performed to prevent reading out of the file boundaries.
Converting a ByteArray to a File
byteArrayToFile( byteArray, filename, optMimeType = 'application/octet-stream' )
This will generate a download of given filename, containing the data of
byteArray as its content using given optMimeType.
To prevent blocking the download, this should be called directly from a click handler.
Example
Let's say we want to read the binary data of a well known proprietary format.
First up we will get to...
Define a structure
Defining a structure is nothing more than declaring an Object where the keys
define names meaningful to your purpose and the values consist of Strings describing:
- one of the available type enumerations (the names of the imported types are equal to their value).
- optional Array declaration where by adding a numerical value between brackets [n], will
make the value an Array of given length n.
- optional modifier defining the endianness of the file's byte order, separated by a pipeline
(either |BE for Big Endian or |LE for Little Endian). When unspecified, the
endianness of the clients system is used (assuming the file has been encoded on/by a similar
system, which usually means Little Endian these days).
An example structure that defines the header of a .WAV file
would look like:
const wavHeader = {
type: 'CHAR[4]',
size: 'INT32|LE',
format: 'CHAR[4]',
formatName: 'CHAR[4]',
formatLength: 'INT32|LE',
audioFormat: 'INT16|LE',
channelAmount: 'INT16|LE',
sampleRate: 'INT32|LE',
bytesPerSecond: 'INT32|LE',
blockAlign: 'INT16|LE',
bitsPerSample: 'INT16|LE',
dataChunkId: 'CHAR[4]',
dataChunkSize: 'INT32|LE'
};
Note that the order of the keys (and more importantly: their type definition) should match
the order of the values as described the particular file's type!
A teeny tiny note on Endianness
Note that specifying endianness can be omitted if you're certain that the file's
encoding is equal to that of the platform you will be parsing the file on (most likely
only Big Endianness will require an explicit definition). And I hope you will never
be in the unfortunate situation where you work with a file that uses different
endianness for different blocks!
Back to talking types
All available data types are listed in the { types } export. Note that definitions
for CHAR will return as a String. If you want an 8-bit integer/byte value, use
BYTE or INT8 instead.
We can now proceed to read the file:
import { parse } from 'binarcular';
async function readWaveHeader( fileReference ) {
const { data, end, error, byteArray } = await parse( fileReference, wavHeader, 0 );
console.log( data ); // will contain the properties of a WAV file header
console.log( end ); // will describe the end offset of the header
console.log( error ); // when true, a file reading error occurred
}
You can also view the demo provided in this repository's example.html file, which
parses .WAV files and provides advanced examples using seeking, slicing and error
correction before finally providing you with the instruction on how to extract the
meaningful data from the file.
Performance
Depending on the size of the files you're working with, memory allocation can become a problem.
The parser will only read the block that is requested (e.g. starting from the
requested offset and only for the size of the requested structureDefinition) and
should thus be light on resources. Additionally, all read operations happen in a
dedicated Web Worker which keeps your main application responsive (you can safely
parse several hundred megabytes of data without blocking your UI).
Depending on your use case, it helps to take the following guidelines into consideration:
- Use base64 only when you have no choice as a base64 String describes the
file in its entirety. Also, the way JavaScript handles Strings is by
allocating the entire value (and not by reference!) whenever you assign
it to a new variable.
- If you intend to do multiple reads on the same file (for instance: first reading
its header to determine where in the file the meaningful content begins) it
is recommended to use the fileToByteArray()-method to create a single
reusable Uint8Array. This also makes sense if you need to read the file in its entirety.
Build instructions
In case you want to aid in development of the library:
The project dependencies are maintained by NPM, you can resolve them using:
npm install
You can develop (and test against the example app by navigating to http://localhost:8080) by running:
npm run dev
To create a production build:
npm run build
After which a folder dist/ is created which contains the prebuilt AMD/RequireJS
and CommonJS/ES module libraries (as well as the example application).
The source code is transpiled to ES5 for maximum browser compatibility.
Unit testing
Unit tests are run via jest, you can run the tests by running:
npm run test
Unit tests go in the ./test-folder. The file name for a unit test must be equal to the file it is testing, but contain the suffix ".spec", e.g. functions.js should have a test file functions.spec.js.