interface-datastore

interface-datastore

Implementation of the datastore interface in JavaScript

Lead Maintainer

Alex Potsides

Table of Contents

• Implementations
• Adapter
• Install
• Usage
  • Wrapping Stores
  • Test suite
  • Keys
• API
  • has(key, [options]) -> Promise<Boolean>
    • Arguments
    • Example
  • put(key, value, [options]) -> Promise
    • Arguments
    • Example
  • putMany(source, [options]) -> AsyncIterator<{ key: Key, value: Buffer }>
    • Arguments
    • Example
  • get(key, [options]) -> Promise<Buffer>
    • Arguments
    • Example
  • getMany(source, [options]) -> AsyncIterator<Buffer>
    • Arguments
    • Example
  • delete(key, [options]) -> Promise
    • Arguments
    • Example
  • deleteMany(source, [options]) -> AsyncIterator<Key>
    • Arguments
    • Example
  • query(query, [options]) -> AsyncIterable<Buffer>
    • Arguments
    • Example
  • batch()
    • Example
    • put(key, value)
    • delete(key)
    • commit([options]) -> Promise<void>
    • Arguments
    • Example
  • open() -> Promise
  • close() -> Promise
• Contribute
• License

Implementations

• Backed Implementations
  • Memory: src/memory
  • level: datastore-level (supports any levelup compatible backend)
  • File System: datstore-fs
• Wrapper Implementations
  • Mount: datastore-core/src/mount
  • Keytransform: datstore-core/src/keytransform
  • Sharding: datastore-core/src/sharding
  • Tiered: datstore-core/src/tiered
  • Namespace: datastore-core/src/namespace

If you want the same functionality as go-ds-flatfs, use sharding with fs.

const FsStore = require('datastore-fs')
const ShardingStore = require('datastore-core').ShardingDatatstore
const NextToLast = require('datastore-core').shard.NextToLast

const fs = new FsStore('path/to/store')

// flatfs now works like go-flatfs
const flatfs = await ShardingStore.createOrOpen(fs, new NextToLast(2))

Adapter

An adapter is made available to make implementing your own datastore easier:

const { Adapter } = require('interface-datastore')

class MyDatastore extends Adapter {
  constructor () {
    super()
  }

  async put (key, val) {
    // your implementation here
  }

  async get (key) {
    // your implementation here
  }

  // etc...
}

See the MemoryDatastore for an example of how it is used.

Install

$ npm install interface-datastore

The type definitions for this package are available on http://definitelytyped.org/. To install just use:

$ npm install -D @types/interface-datastore

Usage

Wrapping Stores

const MemoryStore = require('interface-datastore').MemoryDatastore
const MountStore = require('datastore-core').MountDatastore
const Key = require('interface-datastore').Key

const store = new MountStore({ prefix: new Key('/a'), datastore: new MemoryStore() })

Test suite

Available under src/tests.js

describe('mystore', () => {
  require('interface-datastore/src/tests')({
    async setup () {
      return instanceOfMyStore
    },
    async teardown () {
      // cleanup resources
    }
  })
})

### Aborting requests

Most API methods accept an AbortSignal as part of an options object. Implementations may listen for an abort event emitted by this object, or test the signal.aborted property. When received implementations should tear down any long-lived requests or resources created.

### Concurrency

The streaming (put|get|delete)Many methods are intended to be used with modules such as it-parallel-batch to allow calling code to control levels of parallelisation. The batching method ensures results are returned in the correct order, but interface implementations should be thread safe.

const batch = require('it-parallel-batch')
const source = [{
  key: ..,
  value: ..
}]

// put values into the datastore concurrently, max 10 at a time
for await (const { key, data } of batch(store.putMany(source), 10)) {
  console.info(`Put ${key}`)
}

Keys

To allow a better abstraction on how to address values, there is a Key class which is used as identifier. It's easy to create a key from a Buffer or a string.

const a = new Key('a')
const b = new Key(Buffer.from('hello'))

The key scheme is inspired by file systems and Google App Engine key model. Keys are meant to be unique across a system. They are typically hierarchical, incorporating more and more specific namespaces. Thus keys can be deemed 'children' or 'ancestors' of other keys:

• new Key('/Comedy')
• new Key('/Comedy/MontyPython')

Also, every namespace can be parameterized to embed relevant object information. For example, the Key name (most specific namespace) could include the object type:

• new Key('/Comedy/MontyPython/Actor:JohnCleese')
• new Key('/Comedy/MontyPython/Sketch:CheeseShop')
• new Key('/Comedy/MontyPython/Sketch:CheeseShop/Character:Mousebender')

API

Implementations of this interface should make the following methods available:

has(key, [options]) -> Promise<Boolean>

Check for the existence of a given key

Arguments

Name	Type	Description
key	Key	The key to check the existance of
options	Object	An options object, all properties are optional
options.signal	AbortSignal	A way to signal that the caller is no longer interested in the outcome of this operation

Example

const exists = await store.has(new Key('awesome'))

if (exists) {
  console.log('it is there')
} else {
  console.log('it is not there')
}

put(key, value, [options]) -> Promise

Store a value with the given key.

Arguments

Name	Type	Description
key	Key	The key to store the value under
value	Buffer	Value to store
options	Object	An options object, all properties are optional
options.signal	AbortSignal	A way to signal that the caller is no longer interested in the outcome of this operation

Example

await store.put([{ key: new Key('awesome'), value: Buffer.from('datastores') }])
console.log('put content')

putMany(source, [options]) -> AsyncIterator<{ key: Key, value: Buffer }>

Store many key-value pairs.

Arguments

Name	Type	Description
source	AsyncIterator<{ key: Key, value: Buffer }>	The key to store the value under
value	Buffer	Value to store
options	Object	An options object, all properties are optional
options.signal	AbortSignal	A way to signal that the caller is no longer interested in the outcome of this operation

Example

const source = [{ key: new Key('awesome'), value: Buffer.from('datastores') }]

for await (const { key, value } of store.putMany(source)) {
  console.info(`put content for key ${key}`)
}

get(key, [options]) -> Promise<Buffer>

Arguments

Name	Type	Description
key	Key	The key retrieve the value for
options	Object	An options object, all properties are optional
options.signal	AbortSignal	A way to signal that the caller is no longer interested in the outcome of this operation

Example

Retrieve the value stored under the given key.

const value = await store.get(new Key('awesome'))
console.log('got content: %s', value.toString('utf8'))
// => got content: datastore

getMany(source, [options]) -> AsyncIterator<Buffer>

Arguments

Name	Type	Description
source	AsyncIterator<Key>	One or more keys to retrieve values for
options	Object	An options object, all properties are optional
options.signal	AbortSignal	A way to signal that the caller is no longer interested in the outcome of this operation

Example

Retrieve a stream of values stored under the given keys.

for await (const value of store.getMany([new Key('awesome')])) {
  console.log('got content: %s', value.toString('utf8'))
  // => got content: datastore
}

delete(key, [options]) -> Promise

Delete the content stored under the given key.

Arguments

Name	Type	Description
key	Key	The key to remove the value for
options	Object	An options object, all properties are optional
options.signal	AbortSignal	A way to signal that the caller is no longer interested in the outcome of this operation

Example

await store.delete(new Key('awesome'))
console.log('deleted awesome content :(')

deleteMany(source, [options]) -> AsyncIterator<Key>

Delete the content stored under the given keys.

Arguments

Name	Type	Description
source	AsyncIterator<Key>	One or more keys to remove values for
options	Object	An options object, all properties are optional
options.signal	AbortSignal	A way to signal that the caller is no longer interested in the outcome of this operation

Example

const source = [new Key('awesome')]

for await (const key of store.deleteMany(source)) {
  console.log(`deleted content with key ${key}`)
}

query(query, [options]) -> AsyncIterable<Buffer>

Search the store for some values. Returns an AsyncIterable with each item being a Buffer.

Arguments

Name	Type	Description
query	Object	A query object, all properties are optional
query.prefix	String	Only return values where the key starts with this prefix
query.filters	Array<Function(Buffer) -> Boolean>	Filter the results according to the these functions
query.orders	Array<Function(Array<Buffer>) -> Array<Buffer>>	Order the results according to these functions
query.limit	Number	Only return this many records
query.offset	Number	Skip this many records at the beginning
options	Object	An options object, all properties are optional
options.signal	AbortSignal	A way to signal that the caller is no longer interested in the outcome of this operation

Example

// retrieve __all__ values from the store
let list = []
for await (const value of store.query({})) {
  list.push(value)
}
console.log('ALL THE VALUES', list)

batch()

This will return an object with which you can chain multiple operations together, with them only being executed on calling commit.

Example

const b = store.batch()

for (let i = 0; i < 100; i++) {
  b.put(new Key(`hello${i}`), Buffer.from(`hello world ${i}`))
}

await b.commit()
console.log('put 100 values')

put(key, value)

Queue a put operation to the store.

Name	Type	Description
key	Key	The key to store the value under
value	Buffer	Value to store

delete(key)

Queue a delete operation to the store.

Name	Type	Description
key	Key	The key to remove the value for

commit([options]) -> Promise<void>

Write all queued operations to the underyling store. The batch object should not be used after calling this.

Arguments

Name	Type	Description
options	Object	An options object, all properties are optional
options.signal	AbortSignal	A way to signal that the caller is no longer interested in the outcome of this operation

Example

const batch = store.batch()

batch.put(new Key('to-put'), Buffer.from('hello world'))
batch.del(new Key('to-remove'))

await batch.commit()

open() -> Promise

Opens the datastore, this is only needed if the store was closed before, otherwise this is taken care of by the constructor.

close() -> Promise

Close the datastore, this should always be called to ensure resources are cleaned up.

Contribute

PRs accepted.

Small note: If editing the Readme, please conform to the standard-readme specification.

License

MIT 2017 © IPFS