level-transcoder

level-transcoder

Encode data with built-in or custom encodings. The successor to level-codec that transcodes encodings from and to internal data formats supported by a database. This allows a database to store data in a format of its choice (Buffer, Uint8Array or String) with zero-effort support of known encodings. That includes other encoding interfaces in the ecosystem like codecs, abstract-encoding and multiformats.

Usage

Create a transcoder, passing a desired format:

const { Transcoder } = require('level-transcoder')

const transcoder1 = new Transcoder(['view'])
const transcoder2 = new Transcoder(['buffer'])
const transcoder3 = new Transcoder(['utf8'])

Then select an encoding and encode some data:

// Uint8Array(3) [ 49, 50, 51 ]
console.log(transcoder1.encoding('json').encode(123))

// <Buffer 31 32 33>
console.log(transcoder2.encoding('json').encode(123))

// '123'
console.log(transcoder3.encoding('json').encode(123))

If the Transcoder constructor is given multiple formats then Transcoder#encoding() selects an encoding with the best fitting format. Consider a database like leveldown which has the ability to return data as a Buffer or string. If an encoding.decode(data) function needs a string, we'll want to fetch that data from the database as a string. This avoids the cost of having to convert a Buffer to a string. So we'd use the following transcoder:

const transcoder = new Transcoder(['buffer', 'utf8'])

Then, knowing for example that the return value of JSON.stringify(data) is a UTF-8 string which matches one of the given formats, the 'json' encoding will return a string here:

// '123'
console.log(transcoder.encoding('json').encode(123))

In contrast, data encoded as a 'view' (for now that just means Uint8Array) would get transcoded into the 'buffer' encoding. Copying of data is avoided where possible, like how the underlying ArrayBuffer of a view can be passed to Buffer.from(..) without a copy.

Lastly, encodings returned by Transcoder#encoding() have a format property to be used to forward information to an underlying store. For example: an input value of { x: 3 } using the 'json' encoding which has a format of 'utf8', can be forwarded as value '{"x":3}' with encoding 'utf8'. Vice versa for output.

Encodings

Built-in Encodings

These encodings can be used out of the box and are to be selected by name.

In this table, the input is what encode() accepts. The format is what encode() returns as well as what decode() accepts. The output is what decode() returns. The TypeScript typings of level-transcoder have generic type parameters with matching names: TIn, TFormat and TOut.

Name	Input	Format	Output
'buffer' 1	Buffer, Uint8Array, String	'buffer'	Buffer
'view'	Uint8Array, Buffer, String	'view'	Uint8Array
'utf8'	String, Buffer, Uint8Array	'utf8'	String
'json'	Any JSON type	'utf8'	As input
'hex'	String (hex), Buffer	'buffer'	String (hex)
'base64'	String (base64), Buffer	'buffer'	String (base64)

¹ Aliased as 'binary'. Use of this alias does not affect the ability to transcode.

Transcoder Encodings

It's not necessary to use or reference the below encodings directly. They're listed here for implementation notes and to show how input and output is the same; it's the format that differs.

Custom encodings are transcoded in the same way and require no additional setup. For example: if a custom encoding has { name: 'example', format: 'utf8' } then level-transcoder will create transcoder encodings on demand with names 'example+buffer' and 'example+view'.

Name	Input	Format	Output
'buffer+view'	Buffer, Uint8Array, String	'view'	Buffer
'view+buffer'	Uint8Array, Buffer, String	'buffer'	Uint8Array
'utf8+view'	String, Buffer, Uint8Array	'view'	String
'utf8+buffer'	String, Buffer, Uint8Array	'buffer'	String
'json+view'	Any JSON type	'view'	As input
'json+buffer'	Any JSON type	'buffer'	As input
'hex+view' 1	String (hex), Buffer	'view'	String (hex)
'base64+view' 1	String (base64), Buffer	'view'	String (base64)

¹ Unlike other encodings that transcode to 'view', these depend on Buffer at the moment and thus don't work in browsers if a shim is not included by JavaScript bundlers like Webpack and Browserify.

Ecosystem Encodings

Various modules in the ecosystem, in and outside of Level, can be used with level-transcoder although they follow different interfaces. Common between the interfaces is that they have encode() and decode() methods. The terms "codec" and "encoding" are used interchangeably in the ecosystem. Passing these encodings through Transcoder#encoding() (which is done implicitly when used in an abstract-level database) results in normalized encoding objects as described further below.

Module	Format	Interface	Named
protocol-buffers	buffer	level-codec	❌
charwise	utf8	level-codec	✅
bytewise	buffer	level-codec	✅
lexicographic-integer-encoding	buffer, utf8	level-codec	✅
codecs	buffer	codecs	✅
abstract-encoding	buffer	abstract-encoding	❌
multiformats	view	multiformats	✅
base32-codecs	buffer	codecs	✅

Those marked as not named are modules that export or generate encodings that don't have a name property (or type as an alias). We call these anonymous encodings. They can only be used as objects and not by name. Passing an anonymous encoding through Transcoder#encoding() does give it a name property for compatibility, but the value of name is not deterministic.

API

Transcoder

transcoder = new Transcoder(formats)

Create a new transcoder, providing the formats that are supported by a database (or other). The formats argument must be an array containing one or more of 'buffer', 'view', 'utf8'. The returned transcoder instance is stateful, in that it contains a set of cached encoding objects.

encoding = transcoder.encoding(encoding)

Returns the given encoding argument as a normalized encoding object that follows the level-transcoder encoding interface. The encoding argument may be:

• A string to select a known encoding by its name
• An object that follows one of the following interfaces: level-transcoder, level-codec, codecs, abstract-encoding, multiformats
• A previously normalized encoding, such that encoding(x) equals encoding(encoding(x)).

Results are cached. If the encoding argument is an object and it has a name then subsequent calls can refer to that encoding by name.

Depending on the formats provided to the Transcoder constructor, this method may return a transcoder encoding that translates the desired encoding from / to a supported format. Its encode() and decode() methods will have respectively the same input and output types as a non-transcoded encoding, but its name property will differ.

encodings = transcoder.encodings()

Get an array of encoding objects. This includes:

• Encodings for the formats that were passed to the Transcoder constructor
• Custom encodings that were passed to transcoder.encoding()
• Transcoder encodings for either.

Encoding

data = encoding.encode(data)

Encode data.

data = encoding.decode(data)

Decode data.

encoding.name

Unique name. A string.

encoding.commonName

Common name, computed from name. If this encoding is a transcoder encoding, name will be for example 'json+view' and commonName will be just 'json'. Else name will equal commonName.

encoding.format

Name of the (lower-level) encoding used by the return value of encode(). One of 'buffer', 'view', 'utf8'. If name equals format then the encoding can be assumed to be idempotent, such that encode(x) equals encode(encode(x)).

Encoding Interface

Custom encodings must follow the following interface:

interface IEncoding<TIn, TFormat, TOut> {
  name: string
  format: 'buffer' | 'view' | 'utf8'
  encode: (data: TIn) => TFormat
  decode: (data: TFormat) => TOut
}

Install

With npm do:

npm install level-transcoder

Contributing

Level/transcoder is an OPEN Open Source Project. This means that:

Individuals making significant and valuable contributions are given commit-access to the project to contribute as they see fit. This project is more like an open wiki than a standard guarded open source project.

See the Contribution Guide for more details.

Donate

Support us with a monthly donation on Open Collective and help us continue our work.

License

MIT

Changelog

[1.0.0] - 2021-11-07

:seedling: Initial release.