@omegajs/drive

Omega Drive

@omegajs/drive

See API docs at docs.l1fe.tech

Omega Drive is a secure, real-time distributed file system built for the Omega Network.

Install Via L1FE's NPM

npm config set registry https://npm.l1fe.tech
npm install @omegajs/drive

Install Via L1FE's Git Repository

git clone https://lab.l1fe.tech/omega/drive.git
cd drive
npm install

Usage

const Drive = require('@omegajs/drive')
const Keeper = require('@omegajs/keeper')

const vault = new Keeper('./storage')
const drive = new Drive(vault)

await drive.put('/volume.txt', Buffer.from('example'))
await drive.put('/images/logo.png', Buffer.from('..'))
await drive.put('/images/old-logo.png', Buffer.from('..'))

const buffer = await drive.get('/volume.txt')
console.log(buffer) // => <Buffer ..> "example"

const entry = await drive.entry('/volume.txt')
console.log(entry) // => { seq, key, value: { executable, linkname, volume, metadata } }

await drive.del('/images/old-logo.png')

await drive.symlink('/images/logo.shortcut', '/images/logo.png')

for await (const file of drive.list('/images')) {
  console.log('list', file) // => { key, value }
}

const rs = drive.createReadStream('/volume.txt')
for await (const chunk of rs) {
  console.log('rs', chunk) // => <Buffer ..>
}

const ws = drive.createWriteStream('/volume.txt')
ws.write('new example')
ws.end()
ws.once('close', () => console.log('file saved'))

API

const drive = new Drive(vault, [key])

Creates a new Drive instance. vault must be an instance of Keeper.

By default it uses the scroll at { name: 'db' } from vault, unless you set the public key.

await drive.ready()

Waits until internal state is loaded.

Use it once before reading synchronous properties like drive.discoveryKey, unless you called any of the other APIs.

await drive.close()

Fully close this drive, including its underlying Scroll backed datastructures.

drive.keeper

The Keeper instance used as storage.

drive.db

The underlying ODatabase backing the drive file structure.

drive.scroll

The Scroll used for drive.db.

drive.id

String containing the id (z-base-32 of the public key) identifying this drive.

drive.key

The public key of the Scroll backing the drive.

drive.discoveryKey

The hash of the public key of the Scroll backing the drive.

Can be used as a topic to seed the drive using Flock.

drive.contentKey

The public key of the Volumes instance holding volumes associated with entries in the drive.

drive.writable

Boolean indicating if we can write or delete data in this drive.

drive.readable

Boolean indicating if we can read from this drive. After closing the drive this will be false.

drive.version

Number that indicates how many modifications were made, useful as a version identifier.

drive.supportsMetadata

Boolean indicating if the drive handles or not metadata. Always true.

await drive.put(path, buffer, [options])

Creates a file at path in the drive. options are the same as in createWriteStream.

const buffer = await drive.get(path, [options])

Returns the volume at path in the drive. If no volume exists, returns null.

It also returns null for symbolic links.

options include:

{
  wait: true, // Wait for block to be downloaded
  timeout: 0 // Wait at max some milliseconds (0 means no timeout)
}

const entry = await drive.entry(path, [options])

Returns the entry at path in the drive. It looks like this:

{
  seq: Number,
  key: String,
  value: {
    executable: Boolean, // Whether the volume at path is an executable
    linkname: null, // If entry not symlink, otherwise a string to the entry this links to
    volume: { // Volumes id that can be used to fetch the volume associated with this entry
      blockOffset: Number,
      blockLength: Number,
      byteOffset: Number,
      byteLength: Number
    },
    metadata: null
  }
}

options include:

{
  follow: false, // Follow symlinks, 16 max or throws an error
  wait: true, // Wait for block to be downloaded
  timeout: 0 // Wait at max some milliseconds (0 means no timeout)
}

const exists = await drive.exists(path)

Returns true if the entry at path does exists, otherwise false.

await drive.del(path)

Deletes the file at path from the drive.

const comparison = drive.compare(entryA, entryB)

Returns 0 if entries are the same, 1 if entryA is older, and -1 if entryB is older.

const cleared = await drive.clear(path, [options])

Deletes the volume from storage to free up space, but the file structure reference is kept.

options include:

{
  diff: false // Returned `cleared` bytes object is null unless you enable this
}

const cleared = await drive.clearAll([options])

Deletes all the volumes from storage to free up space, similar to how drive.clear() works.

options include:

{
  diff: false // Returned `cleared` bytes object is null unless you enable this
}

await drive.purge()

Purge both scrolls (db and volumes) from your storage, completely removing all the drive's data.

await drive.symlink(path, linkname)

Creates an entry in drive at path that points to the entry at linkname.

If a volume entry currently exists at path then it will get overwritten and drive.get(key) will return null, while drive.entry(key) will return the entry with symlink information.

const batch = drive.batch()

Useful for atomically mutate the drive, has the same interface as Drive.

await batch.flush()

Commit a batch of mutations to the underlying drive.

const stream = drive.list(folder, [options])

Returns a stream of all entries in the drive at paths prefixed with folder.

options include:

{
  recursive: true | false // Whether to descend into all subfolders or not
}

const stream = drive.readdir(folder)

Returns a stream of all subpaths of entries in drive stored at paths prefixed by folder.

const stream = await drive.entries([range], [options])

Returns a read stream of entries in the drive.

options are the same as ODatabase().createReadStream([range], [options]).

const mirror = drive.mirror(out, [options])

Efficiently mirror this drive into another. Returns a MirrorDrive instance constructed with options.

Call await mirror.done() to wait for the mirroring to finish.

const watcher = db.watch([folder])

Returns an iterator that listens on folder to yield changes, by default on /.

Usage example:

for await (const [current, previous] of watcher) {
  console.log(current.version)
  console.log(previous.version)
}

Those current and previous are snapshots that are auto-closed before next value.

Don't close those snapshots yourself because they're used internally, let them be auto-closed.

await watcher.ready()

Waits until the watcher is loaded and detecting changes.

await watcher.destroy()

Stops the watcher. You could also stop it by using break in the loop.

const rs = drive.createReadStream(path, [options])

Returns a stream to read out the volume stored in the drive at path.

options include:

{
  start: Number, // `start` and `end` are inclusive
  end: Number,
  length: Number, // `length` overrides `end`, they're not meant to be used together
  wait: true, // Wait for blocks to be downloaded
  timeout: 0 // Wait at max some milliseconds (0 means no timeout)
}

const ws = drive.createWriteStream(path, [options])

Stream a volume into the drive at path.

options include:

{
  executable: Boolean,
  metadata: null // Extended file information i.e. arbitrary JSON value
}

await drive.download(folder, [options])

Downloads the volumes corresponding to all entries in the drive at paths prefixed with folder.

options are the same as those for drive.list(folder, [options]).

const snapshot = drive.checkout(version)

Get a read-only snapshot of a previous version.

const stream = drive.diff(version, folder, [options])

Efficiently create a stream of the shallow changes to folder between version and drive.version.

Each entry is sorted by key and looks like this:

{
  left: Object, // Entry in folder at drive.version for some path
  right: Object, // Entry in folder at drive.checkout(version) for some path
}

If an entry exists in drive.version of the folder but not in version, then left is set and right will be null, and vice versa.

await drive.downloadDiff(version, folder, [options])

Downloads all the volumes in folder corresponding to entries in drive.checkout(version) that are not in drive.version.

In other words, downloads all the volumes added to folder up to version of the drive.

await drive.downloadRange(dbRanges, volumeRanges)

Downloads the entries and volumes stored in the ranges dbRanges and volumeRanges.

const done = drive.findingPeers()

Indicate to Drive that you're finding peers in the background, requests will be on hold until this is done.

Call done() when your current discovery iteration is done, i.e. after flock.flush() finishes.

const stream = drive.replicate(isInitiatorOrStream)

Usage example:

const flock = new Flock()
const done = drive.findingPeers()
flock.on('connection', (socket) => drive.replicate(socket))
flock.join(drive.discoveryKey)
flock.flush().then(done, done)

See more about how replicate works at keeper.replicate.

const updated = await drive.update([options])

Waits for initial proof of the new drive version until all findingPeers are done.

options include:

{
  wait: false
}

Use drive.findingPeers() or { wait: true } to make await drive.update() blocking.

const volumes = await drive.getVolumes()

Returns the Volumes instance storing the volumes indexed by drive entries.

await drive.put('/file.txt', Buffer.from('hi'))

const buffer1 = await drive.get('/file.txt')

const volumes = await drive.getVolumes()
const entry = await drive.entry('/file.txt')
const buffer2 = await volumes.get(entry.value.volume)

// => buffer1 and buffer2 are equals

License

Apache-2.0