block-cache

block-cache

block-cache is a transparent(ish) cache that keeps data split in blocks in an in-memory lru-cache. This is useful if you want to process a file, reusing previously downloaded parts and improving the general performance without caching more than your given memory limit.

npm i block-cache --save

Usage

The API of block-cache is comparable to the fs API but all callbacks are optional and if omitted will result in a promise returned.

Here is a simple example of reading a file into the local cache.

const fs = require('fs')
const {Cache, CachedFile} = require('block-cache')

const cache = new Cache(fs, {
  blkSize: 1024,
  cacheSize: 2 * 1024 * 1024 // 2 MB
})
const fp = await cache.open('./Readme.md')
const data = await cache.read(fp)

console.log(data)

await cache.close(fp)

This example reads the entirety of the ./Readme.md file into a 2 mega-byte cache in 1 kilo-byte sized blocks and then closes the data. Even if the fp is closed: the block stay in the cache!

Use-case: file parsing

This library usually comes in play when you have to parse parts of a file depending on the header. Take the beginning of this GIF parser for example:

const fs = require('fs')
const {Cache, CachedFile} = require('block-cache')

const cache = new Cache(fs, {
  blkSize: 1024,
  cacheSize: 2 * 1024 * 1024 // 2 MB
})
const fp = await cache.open('./Readme.md')
const signature = (await fp.read(null, 0, 6)).toString()
if (signature === 'GIF87a' || signature === 'GIF89a') {
  const packed = await fp.read(null, 0, 10)
  // etc.
}

await cache.close(fp)

As you can see in this example code, it is necessary to read only parts of a file at a time. Very small parts. But most of those bytes are already present in the cache. So, while the first operation needed to read 1Kb of the file, the second operation can already use it from the cached data.

API

• Cache
  • .open
  • .openSync
  • .read
  • .createReadStream
• CachedFile
  • .read
  • .createReadStream
  • .fd
  • .size
  • .stat
  • .prefix
  • DEFAULT_BLK_SIZE

---

new Cache(fs[, opts])

• fs is a FileSystem (require('fs'))) or Hyperdrive archive (object).
• opts.cache is a lru-cache instance (object, optional).
• opts.cacheSize is the size of the lru-cache to be created in case a opts.cache is missing. 10MB by default (integer).
• opts.blkSize is the default size in bytes of a cache-block. Defaults to CachedFile.DEFAULT_BLK_SIZE. (integer).

---

cache.open(path[, opts, cb])

Creates a cached file pointer reference for a given path. Note: It will open the file reference in r mode.

• path path to read the file from (string).
• opts.blkSize is the size in bytes of a cache-block. Defaults to the opts.blkSize defined in the Cache.
• cb(Error, CachedFile) is an optional async callback handler method. The method will return a Promise if the callback is not defined.

---

cache.openSync(path[, opts])

like cache.open but synchronous.

---

cache.read(fd[, buffer, offset, length, position, cb])

Reads the content of an opened file into a given buffer.

• fd is a CachedFile instance, created with .open or .openSync
• buffer is a Buffer instance to write into. Unlike the Node API, this is optional which means that the reader will create a buffer instance if null or undefined is passed-in.
• offset is the offset in the buffer to start writing at.
• length is an integer specifying the number of bytes to read into buffer, defaults to length of the file (integer).
• position is an argument specifying where to begin reading from in the file. The file descriptor will remember the end of the last read in the fd.position property. It will default to 0.
• cb(Error, Buffer) is an optional async callback handler method. The method will return a Promise if the callback is not defined.

---

cache.createReadStream(path[, opts, cb])

Creates a cached file pointer reference for a given path and then reads it through a stream.

• path is the path to read the file from (string).
• opts.blkSize is the block size for each block to be cached. Defaults to CachedFile.DEFAULT_BLK_SIZE. (integer).
• opts.start is the start from while to read the file. Defaults to 0. (integer)
• opts.end is the end until which to read the file. Defaults to the end of the file. (integer)

---

new CachedFile(cache, path[, opts])

Creates a new instance for reading one file. The blocks will still be stored in the passed-in cache object.

• cache is a Cache instance.
• path is the path to read the file from (string).
• opts.blkSize specifies the block size for this file pointer (integer). Defaults to cache.opts.blkSize or to CachedFile.DEFAULT_BLK_SIZE.

---

cachedFile.read([buffer, offset, length, position, cb])

Like cache.read but without the need to pass a descriptor.

---

cachedFile.createReadStream([opts, cb])

Like cache.createReadStream but without the need to pass a descriptor.

---

cachedFile.fd([cb])

Retreives the actual file descriptor for that path on the file system.

---

cachedFile.size([cb])

The size of the file as noted in the file descriptor.

---

cachedFile.prefix([cb])

The prefix for ranges of the file stored in cache.

---

cachedFile.stat([cb])

Retreives the actual Stats of the file through fs.stat.

---

CachedFile.DEFAULT_BLK_SIZE

The default blk size used for caching.

License

MIT