level-js

level-js

An abstract-leveldown compliant store on top of IndexedDB, which is in turn implemented on top of LevelDB which brings this whole shebang full circle.

Table of Contents

Click to expand

• Background
• Example
• Browser Support
• Type Support
• Install
• API
• Running Tests
• Big Thanks
• Contributing
• Donate
• License

Background

Here are the goals of level-js:

• Store large amounts of data in modern browsers
• Pass the full abstract-leveldown test suite
• Support Buffer keys and values
• Support all key types of IndexedDB Second Edition
• Support all value types of the structured clone algorithm except for null and undefined
• Be as fast as possible
• Sync with multilevel over ASCII or binary transports.

Being abstract-leveldown compliant means you can use many of the Level modules on top of this library. For some demos of it working, see @brycebaril's presentation Path of the NodeBases Jedi.

Example

If you are upgrading: please see UPGRADING.md.

var levelup = require('levelup')
var leveljs = require('level-js')
var db = levelup(leveljs('bigdata'))

db.put('hello', Buffer.from('world'), function (err) {
  if (err) throw err

  db.get('hello', function (err, value) {
    if (err) throw err

    console.log(value.toString()) // 'world'
  })
})

In ES6 browsers:

const levelup = require('levelup')
const leveljs = require('level-js')
const db = levelup(leveljs('bigdata'))

await db.put('hello', Buffer.from('world'))
const value = await db.get('hello')

Browser Support

Type Support

Unlike leveldown, level-js does not stringify keys or values. This means that in addition to strings and Buffers you can store almost any JavaScript type without the need for encoding-down.

Values

All value types of the structured clone algorithm are supported except for null and undefined. Depending on the environment, this includes:

• Number, including NaN, Infinity and -Infinity
• String, Boolean, Date, RegExp, Array, Object
• ArrayBuffer or a view thereof (typed arrays);
• Map, Set, Blob, File, FileList, ImageData (limited support).

In addition level-js stores Buffer values without transformation. This works in all target environments because Buffer is a subclass of Uint8Array, meaning such values can be passed to IndexedDB as-is.

When getting or iterating binary values, regardless of whether they were stored as a Buffer, ArrayBuffer or a view thereof, values will return as a Buffer. This behavior can be disabled, in which case ArrayBuffer returns as ArrayBuffer, typed arrays return as typed arrays and Buffer returns as Uint8Array:

db.get('key', { asBuffer: false })
db.iterator({ valueAsBuffer: false })

If the environment does not support a type, it will throw an error which level-js catches and passes to the callbacks of put or batch. For example, IE does not support typed array values. At the time of writing, Chrome is the only browser that supports all types listed above.

Keys

All key types of IndexedDB Second Edition are supported. Depending on the environment, this includes:

• Number, including Infinity and -Infinity, but not NaN
• Date, except invalid (NaN)
• String
• ArrayBuffer or a view thereof (typed arrays);
• Array, except cyclical, empty and sparse arrays. Elements must be valid types themselves.

In addition you can use Buffer keys, giving level-js the same power as implementations like leveldown and memdown. When iterating binary keys, regardless of whether they were stored as Buffer, ArrayBuffer or a view thereof, keys will return as a Buffer. This behavior can be disabled, in which case binary keys will always return as ArrayBuffer:

db.iterator({ keyAsBuffer: false })

Note that this behavior is slightly different from values due to the way that IndexedDB works. IndexedDB stores binary values using the structured clone algorithm, which preserves views, but it stores binary keys as an array of octets, so that it is able to compare and sort differently typed keys.

If the environment does not support a type, it will throw an error which level-js catches and passes to the callbacks of get, put, del, batch or an iterator. Exceptions are:

• null and undefined: rejected early by abstract-leveldown
• Binary and array keys: if not supported by the environment, level-js falls back to String(key).

Normalization

If you desire normalization for keys and values (e.g. to stringify numbers), wrap level-js with encoding-down. Alternatively install level-browserify which conveniently bundles levelup, level-js and encoding-down. Such an approach is also recommended if you want to achieve universal (isomorphic) behavior or to smooth over type differences between browsers. For example, you could have leveldown in a backend and level-js in the frontend.

Another reason you might want to use encoding-down is that the structured clone algorithm, while rich in types, can be slower than JSON.stringify.

Sort Order

Unless level-js is wrapped with encoding-down, IndexedDB will sort your keys in the following order:

1. number (numeric)
2. date (numeric, by epoch offset)
3. binary (bitwise)
4. string (lexicographic)
5. array (componentwise).

You can take advantage of this fact with levelup streams. For example, if your keys are dates, you can select everything greater than a specific date (let's be happy and ignore timezones for a moment):

const db = levelup(leveljs('time-db'))

db.createReadStream({ gt: new Date('2019-01-01') })
  .pipe(..)

Or if your keys are arrays, you can do things like:

const db = levelup(leveljs('books-db'))

await db.put(['Roald Dahl', 'Charlie and the Chocolate Factory'], {})
await db.put(['Roald Dahl', 'Fantastic Mr Fox'], {})

// Select all books by Roald Dahl
db.createReadStream({ gt: ['Roald Dahl'], lt: ['Roald Dahl', '\xff'] })
  .pipe(..)

To achieve this on other abstract-leveldown implementations, wrap them with encoding-down and charwise (or similar).

Known Browser Issues

IE11 and Edge yield incorrect results for { gte: '' } if the database contains any key types other than strings.

Buffer vs ArrayBuffer

For interoperability it is recommended to use Buffer as your binary type. While we recognize that Node.js core modules are moving towards supporting ArrayBuffer and views thereof, Buffer remains the primary binary type in the Level ecosystem.

That said: if you want to put() an ArrayBuffer you can! Just know that it will come back as a Buffer by default. If you want to get() or iterate stored ArrayBuffer data as an ArrayBuffer, you have a few options. Without encoding-down:

const db = levelup(leveljs('mydb'))

// Yields an ArrayBuffer, Buffer and ArrayBuffer
const value1 = await db.get('key', { asBuffer: false })
const value2 = await db.get('key')
const value3 = value2.buffer

With encoding-down (or level-browserify) you can use the id encoding to selectively bypass encodings:

const encode = require('encoding-down')
const db = levelup(encode(leveljs('mydb'), { valueEncoding: 'binary' }))

// Yields an ArrayBuffer, Buffer and ArrayBuffer
const value1 = await db.get('key', { valueEncoding: 'id' })
const value2 = await db.get('key')
const value3 = value2.buffer

Install

With npm do:

npm install level-js

Not to be confused with leveljs.

This library is best used with browserify.

API

db = leveljs(location[, options])

Returns a new leveljs instance. location is the string name of the IDBDatabase to be opened, as well as the object store within that database. The database name will be prefixed with options.prefix.

options

The optional options argument may contain:

• prefix (string, default: 'level-js-'): Prefix for IDBDatabase name.
• version (string | number, default: 1): The version to open the database with.

See IDBFactory#open for more details.

Running Tests

git clone [e-mail]:Level/level-js.git
cd level-js
npm install
npm test

It will print out a URL to open in a browser of choice.

Big Thanks

Cross-browser Testing Platform and Open Source ♥ Provided by Sauce Labs.

Contributing

Level/level-js is an OPEN Open Source Project. This means that:

Individuals making significant and valuable contributions are given commit-access to the project to contribute as they see fit. This project is more like an open wiki than a standard guarded open source project.

See the Contribution Guide for more details.

Donate

To sustain Level and its activities, become a backer or sponsor on Open Collective. Your logo or avatar will be displayed on our 28+ GitHub repositories, npm packages and (soon) our website. 💖

Backers

Sponsors

License

MIT © 2012-present Max Ogden and Contributors.