flash-store

flash-store

FlashStore is Key-Value persistent storage with easy to use ES6 Map-like API(both Async and Sync support), powered by LevelDB and TypeScript.

Requirements

• Node.js v10 or above

Examples

Try the following command

npm install
npm run demo

The basic function as follows:

import { FlashStore } from 'flash-store'

const flashStore = new FlashStore('flashstore.workdir')

await flashStore.set(1, 'a')
console.log(`key: 1, value: ${await flashStore.get(1)}`)
// Output: 'a'

await flashStore.del(1)
console.log(`key: 1, value: ${await flashStore.get(1)}`)
// Output: undefined

Supported Backend

Backend	Flash Store	Install NPM Command
LevelDB	v1.0	npm install flash-store@1
SQLite	v0.20	npm install flash-store@0.20
Medea	v0.18	npm install flash-store@0.18 (deprecated)
SnapDB	v0.16	npm install flash-store@0.16 (deprecated)
RocksDB	v0.14	npm install flash-store@0.14
LevelDB	v0.12	npm install flash-store@0.12

SnapDB & Medea were all deprecated because of lots of unknown bugs.

API Reference

FlashStore

FlashStore implements the Standard ES6 Map API with Async modification, powered by async-map-like

/**
 * ES6 Map API with Async
 */
export interface AsyncMap<K = any, V = any> {
  [Symbol.asyncIterator]() : AsyncIterableIterator<[K, V]>
  size                     : Promise<number>

  clear   ()                 : Promise<void>
  delete  (key: K)           : Promise<void>
  entries()                  : AsyncIterableIterator<[K, V]>
  get     (key: K)           : Promise<V | undefined>
  has     (key: K)           : Promise<boolean>
  keys    ()                 : AsyncIterableIterator<K>
  set     (key: K, value: V) : Promise<void>
  values  ()                 : AsyncIterableIterator<V>
}

class FlashStore<K, V> implements AsyncMap<K, V> {}

FlashStoreSync

FlashStoreSync implements the Standard ES6 Map API:

class FlashStoreSync<K, V> implements Map<K, V> {}

• You get a sync API at the cost of all your data have to be kept in memory.
• The data will be async writing back to disk for persistent storage in background.
• The performance of FlashStoreSync can be expected high because it's all in memory.

Document

See auto generated docs

• ECMAScript 6: Maps - 2ality – JavaScript and more

See Also

• Node.js databases: an embedded database using LevelDB
• How to Cook a Graph Database in a Night - LevelGraph
• Graph database JS style for Node.js and the Browser. Built upon LevelUp and LevelDB.
• 浅析 BigTable 和 LevelDB 的实现

Known Issues

• The gte and lte in options do not work property. (#4)

Version History

master

v1.0 (Aug 16, 2021) LevelDB v7.0

LevelDB v7.0.0 support.

v0.20 Apr 2020 SQLite as Backend

• We hardcoded the key type to be string only in this version.
• We decide to try better-sqlite3 as it claim is very fast.
• The other alternates (would love to try in the future if necessary):
  • TypeScript: sqlite
  • WebAssembly: sql.js

v0.18 Feb 2019 - Medea as Backend

DEPRECATED: Due to #79 #74 and also it is very unstable in production as my experiences. (e.g. memory leak & block event loop)

• Switch from SnapDB to MedeaDown

Medea is a persistent key-value storage library that runs everywhere Node runs.

"It is a pure JS implementation of leveldown and it's almost as fast." — @Raynos link

"The LevelDOWN-compatible wrapper for Medea really opens up the possibility to reuse the modules that have already been created around the LevelUP ecosystem." — @kevinswiber link

Known Issues: FlashStore 会写满磁盘的问题 #155

async function compact (store: FlashStore): Promise<void> {
  await store.size
  const db = (store as any).levelDb.db.db.db
  await new Promise((resolve, reject) => {
    db.compact((err: any) => {
      if (err) {
        return reject(err)
      }
      resolve()
    })
  })
}

v0.16 May 2019 - SnapDB as Backend

• Switch from RocksDB to SnapDB #45
• #50 has been fixed. WARN: Do not use this version because it has known issues

v0.14 May 2019 - RocksDB as Backend

• Switch from LevelDB to RocksDB #34

v0.12 Jan 2019 - LevelDB as Backend

• Use LevelDB as backend to skip the compiling when install.
• Using leveldb official typings from @types/

v0.7 Aug 2018 - Nosql-LevelDB as Backend

• Use nosql-leveldb as backend to prevent segfault.

v0.6 Jul 2018

• Upgrade to TypeScript 3.0

v0.4 Jun 2018

1. Refactor API to implement ES6 Map interface

• Update the API to ES6 Map-like, the difference is that FlashStore is all async.

2. Add FlashStoreSync as a in-memory Write-back Cache for Flash-Store

Add a new class FlashStoreSync which is a in-memory full loaded Write-back Cache for Flash-Store:

• Writes directly to cache, lazy writes to store.
• Reads from cache, never read-miss because cache have the full data of the store which will never expire.
• API of FlashStoreSync is the same as the ES6 Map

v0.2 Sep 2017

Init version, API is LevelDB-like.

FAQ

Q: What's the difference between the flash-store and memory-card

Short answer:

• flash-store is for save data to local filesystem.
• memory-card is for save data to a distributed network storage, it can be serialized/deserialized safely by design.

Long answer:

flash-store and memory-card are all built by @huan, and they are all follow the ES6 Map API.

flash-store is using a no-sql local file database to maximum the performance, it can be used as a local database, or a local cache for whatever you want to cache from other API.

memory-card is using a local file to store data in JSON format by default, however, it supports more distributed methods. Learn more from it's repository at here

Author

Huan LI (李卓桓) [e-mail]

Copyright & License

• Code & Docs © 2017-now Huan (李卓桓) <[e-mail]>
• Code released under the Apache-2.0 License
• Docs released under Creative Commons