block-cache

block-cache

block-cache is a transparent(ish) cache that keeps data split in blocks in an in-memory lru-cache. This is useful if you want to process a file, reusing previously downloaded parts and improving the general performance without caching more than your given memory limit.

The cache does not expose the passed-in API at any point which makes it suitable as a Sandbox.

npm i block-cache --save

Usage

The API of block-cache is comparable to the fs API but all callbacks are optional and if omitted will result in a Promise returned.

Here is a simple example of reading a file into the local cache.

const fs = require('fs')
const {Cache, CachedFile} = require('block-cache')

const cache = new Cache(fs, {
  blkSize: 1024,
  cacheSize: 2 * 1024 * 1024 // 2 MB
})
const fp = await cache.open('./Readme.md')
const data = await cache.read(fp)

console.log(data)

await cache.close(fp)

This example reads the entirety of the ./Readme.md file into a 2 mega-byte cache in 1 kilo-byte sized blocks and then closes the data. Even if the fp is closed: the block stay in the cache!

Use-case: file parsing

This library usually comes in play when you have to parse parts of a file depending on the header. Take the beginning of this GIF parser for example:

const fs = require('fs')
const {Cache, CachedFile} = require('block-cache')

const cache = new Cache(fs, {
  blkSize: 1024,
  cacheSize: 2 * 1024 * 1024 // 2 MB
})
const fp = await cache.open('./Readme.md')
const signature = (await fp.read(null, 0, 6)).toString()
if (signature === 'GIF87a' || signature === 'GIF89a') {
  const packed = await fp.read(null, 0, 10)
  // etc.
}

await cache.close(fp)

As you can see in this example code, it is necessary to read only parts of a file at a time. Very small parts. But most of those bytes are already present in the cache. So, while the first operation needed to read 1Kb of the file, the second operation can already use it from the cached data.

API

• Cache
  • .open
  • .close
  • .disconnect
  • .openSync
  • .read
  • .createReadStream
  • DEFAULT_CACHE_SIZE
• CachedFile
  • .close
  • .read
  • .createReadStream
  • .size
  • .stat
  • DEFAULT_BLK_SIZE

---

new Cache(fs[, opts])

• fs is a FileSystem (require('fs'))) or Hyperdrive archive (object).
• opts.cache is a lru-cache instance (object, optional).
• opts.cacheSize is the size of the lru-cache to be created in case a opts.cache is missing. Defaults to Cache.DEFAULT_CACHE_SIZE (integer).
• opts.blkSize is the default size in bytes of a cache-block. Defaults to CachedFile.DEFAULT_BLK_SIZE. (integer).
• opts.prefix is an optional prefix that can be added to the cached data, useful if you want to reuse the same opts.cache for multiple Cache instances. Defaults to ''. (string)

---

cache.open(path[, opts, cb])

Creates a cached file pointer reference for a given path. Note: It will open the file reference in r mode.

• path path to read the file from (string).
• opts.blkSize is the size in bytes of a cache-block. Defaults to the opts.blkSize defined in the Cache.
• cb(Error, CachedFile) is an optional async callback handler method. The method will return a Promise if the callback is not defined.

---

cache.close(fp[, cb])

Closes a created file pointer reference. After closing, future requests on the CachedFile will result in an err.code === 'ERR_CLOSED error.

• fp is a CachedFile instance, created with .open or .openSync
• cb(Error) is an optional async callback handler method. The method will return a Promise if the callback is not defined.

---

cache.disconnect()

Disconnects the cache from the file system instance. Any future operations on the Cache or CachedFile instances create with the Cache will result in an err.code === 'ERR_DISCONNECTED' error. Disconnect also closes all open file pointer references on the underlying file system.

---

cache.openSync(path[, opts])

like cache.open but synchronous.

---

cache.read(fd[, buffer, offset, length, position, cb])

Reads the content of an opened file into a given buffer.

• fd is a CachedFile instance, created with .open or .openSync
• buffer is a Buffer instance to write into. Unlike the Node API, this is optional which means that the reader will create a buffer instance if null or undefined is passed-in.
• offset is the offset in the buffer to start writing at.
• length is an integer specifying the number of bytes to read into buffer, defaults to length of the file (integer).
• position is an argument specifying where to begin reading from in the file. The file descriptor will remember the end of the last read in the fd.position property. It will default to 0.
• cb(Error, Buffer) is an optional async callback handler method. The method will return a Promise if the callback is not defined.

---

cache.createReadStream(path[, opts, cb])

Creates a cached file pointer reference for a given path and then reads it through a stream.

• path is the path to read the file from (string).
• opts.blkSize is the block size for each block to be cached. Defaults to cache.opts.blkSize. (integer).
• opts.start is the start from while to read the file. Defaults to 0. (integer)
• opts.end is the end until which to read the file. Defaults to the end of the file. (integer)

Cache.DEFAULT_CACHE_SIZE

The default size of a cache created if opts.cache is not passed in: 10485760 (integer, equals 10 MegaByte)

---

new CachedFile(cache, path[, opts])

Creates a new instance for reading one file. The blocks will still be stored in the passed-in cache object. While it is possible to instantiate a new CachedFile, you can not pass-in a cache directly, use the .open, .openSync or .createReadStream to interact with the cache

• cacheInternal a subset of the Cache API that is not accessible from outside.
• cacheInternal.open(path, opts, cb) opens a file pointer to a given path on the underlying fs.
• cacheInternal.stat(path, cb) receives the stat file from the underlying fs
• cacheInternal.close(fp, cb) closes a file pointer on the underlying fs.
• cacheInternal.read(fp, prefix, start, end, cb) reads bytes from the underlying fs into a buffer.
• opts.blkSize specifies the block size for this file pointer (integer). Defaults to CachedFile.DEFAULT_BLK_SIZE.

---

cachedFile.close([cb])

Closes the instance. After closing, future requests on the CachedFile will result in an err.code === 'ERR_CLOSED error.

• cb(Error) is an optional async callback handler method. The method will return a Promise if the callback is not defined.

---

cachedFile.read([buffer, offset, length, position, cb])

Like cache.read but without the need to pass a descriptor.

---

cachedFile.createReadStream([opts, cb])

Like cache.createReadStream but without the need to pass a descriptor.

---

cachedFile.size([cb])

The size of the file as noted in the file descriptor.

---

cachedFile.stat([cb])

Retreives the actual Stats of the file through fs.stat.

---

CachedFile.DEFAULT_BLK_SIZE

The default opts.blkSize used for caching: 512 (integer, equals 512 Byte).

Acknowledgement

This project was made for and supported by dotloom.

License

MIT

Changelog

2.0.0 (2018-03-06)

Features

• api: Added disconnect api that allows the disconnection of a Cache instance from its fs (f87135e)
• api: Exposed DEFAULT_CACHE_SIZE (10MB) in a tested fashion (c2f6dbd)
• api: New option prefix on Cache allows reuse of underlying lru-cache. (2862257)
• freeze: file pointers created through Cache.open and Cache.openSync are frozen now. (e5a00ce)
• freeze: Using defineProperties for CachedFile properties to make sure that the instances are freezable. (f3222c1)
• sandbox: Sandboxing the Cache to make sure users of Cache can not access/modify the filesystem. (ac90063)

BREAKING CHANGES

• sandbox: The documented method Cached.fd and Cached.prefix were giving informations about the implementation details and they have been removed.

<a name="1.0.0"></a>