@nrk/nodecache-as-promised

@nrk/nodecache-as-promised

Fast and resilient cache for NodeJs targeting high-volume sites

• Installing
• Features
• APIs
• Examples
• Distributed capabilites
• Local development
• Creating your own middleware
• Building and committing

Installing

npm install @nrk/nodecache-as-promised --save

Motivation

Sometimes Node.js needs to do some heavy lifting, performing CPU or network intensive tasks and yet respond quickly on incoming requests. For repetitive tasks like Server side rendering of markup or parsing big JSON responses caching can give the application a great performance boost. Since many requests may hit the server concurrently, you do not want more than one worker to run for a given resource at the same time. In addition - serving stale content when a backend resource is down may save your day! The intention of nodecache-as-promised is to give you a fairly simple, yet powerful application cache, with fine-grained control over caching behaviour.

nodecache-as-promised is inspired by how Varnish works. It is not intended to replace Varnish (but works great in combination). In general Varnish works great as an edge/burst/failover cache, in addition to reverse proxying and loadbalancing. There exists several other cache solutions on NPM, but they're often too basic or too attached to a combination of perquisites that does not fit all needs of an application cache.

Features

• In-memory cache is used as primary storage since it will always be faster than parsing and fetching data over network. An LRU-cache is enabled to constrain the amount of memory used.
• Caches are filled using worker promises since cached objects often are depending on async operations. (RxJs)[https://www.npmjs.com/package/rxjs] is used to queue concurrent requests for the same key; thus ensuring that only one worker is performed when cached content is missing/stale.
• Caching of custom class instances, functions and native objects such as Date, RegExp and redux stores are supported through in-memory caching. Non-serializable (using JSON.stringify) objects are filtered out in persistent caches though.
• Grace mode is used if a worker promise fails (eg. caused by failing backends), ie. stale cache is returned instead.
• Avoidance of spamming backend resources using a configurable retry-wait parameter, serving either a stale object or rejection.
• Middleware support so you may create your own custom extensions. Provided middlewares:
  • Persistent cache is used as secondary storage to avoid high back-pressure on backend resources when caches are cleared after server restarts. This is achieved storing cache-misses in Redis depending on a ioredis-factory
  • Distributed on demand expiry so that new content may be published across servers/instances before cache-TTL is reached. This is achieved using Redis pub/sub depending on a ioredis-factory

Performance testing

Parsing a json-file at around 47kb (file contents are cached at startup). Using a Macbook pro, mid 2015, 16gb ram, i7 CPU.

The image shows a graph from running the test script npm run perf:nocache-cache-file -- --type=linear. At around 1300 iterations the event loop starts lagging, and at around 1500 iterations the process stops responding. It displays that even natively optimized JSON.parse could be a bottleneck when fetching remote API-data for rendring. (React.render would be even slower)

The second image is a graph from running test script npm run perf:cache -- --type=linear. At around 3.1 million iterations the event loop starts lagging, and at around 3.4 million iterations the process runs out of memory and crashes. The graph has no relation to how fast JSON.parse is, but what speed is achievable by skipping it altogether (ie. Promise-processing)

APIs

Create a new inMemoryCache instance using a factory method. This instance may be extended by the distCache and/or persistentCache middlewares (.use(..)).

inMemoryCache factory

Creating a new instance

import inMemoryCache from '@nrk/nodecache-as-promised'
const cache = inMemoryCache(options)

options

An object containing configuration

• initial - Object. Initial key/value set to prefill cache. Default: {}
• maxLength - Number. Max key count before LRU-cache evicts object. Default: 1000
• maxAge - Number. Max time before a (stale) key is evicted by LRU-cache (in ms). Default: 172800000 (48h)
• log - Object with log4j-facade. Used to log internal work. Default: console

distCache factory

Creating a new distCache middleware instance

import cache, {distCache} from '@nrk/nodecache-as-promised'
const cache = inMemoryCache()
cache.use(distCache(redisFactory, namespace))

Parameters

Parameters that must be provided upon creation:

• redisFactory - Function. A function that returns an ioredis compatible redisClient.
• namespace - String. Pub/sub-namespace used for distributed expiries

persistentCache factory

Creating a new persistentCache middleware instance

import cache, {persistentCache} from '@nrk/nodecache-as-promised'
const cache = inMemoryCache()
cache.use(persistentCache(redisFactory, options))

Parameters

Parameters that must be provided upon creation:

• redisFactory - Function. A function that returns an ioredis compatible redisClient.

options

• doNotPersist - RegExp. Keys matching this regexp is not persisted to cache. Default null
• keySpace - String. Prefix used when storing keys in redis.
• grace - Number. Used to calculate TTL in redis (before auto removal), ie. object.TTL + grace. Default 86400000 (24h)
• bootload - Boolean. Flag to choose if persisted cache is loaded from redis on middleware creation. Default true

Instance methods

When the factory is created (with or without middlewares), the following methods may be used.

.get(key, [options])

Get an item from the cache.

const {value} = cache.get('myKey')
console.log(value)

Using parameter options - the function either fetches a value from cache or executes provided worker if the cache is stale or cold. The worker will set the cache key if ran and thus returns a Promise

cache.get('myKey', options)
  .then(({value}) => {
    console.log(value)
  })

options

Configuration for the newly created object

• worker - function. A function that returns a promise which resolves new value to be set in cache.
• ttl - Number. Ttl (in ms) before cached object becomes stale. Default: 86400000 (24h)
• workerTimeout - Number. max time allowed to run promise. Default: 5000
• deltaWait - Number. delta wait (in ms) before retrying promise, when stale. Default: 10000

NOTE: It might seem a bit strange to set cache values using .get - but it is to avoid a series of operations using .get() to check if a value exists, then call .set(), and finally running .get() once more (making queing difficult). In summary: .get() returns a value from cache or a provided worker.

.set(key, value, [ttl])

Set a new cache value.

// set a cache value that becomes stale after 1 minute
cache.set('myKey', 'someData', 60 * 1000)

If ttl-parameter is omitted, a default will be used: 86400000 (24h)

.expire(keys)

Mark keys as stale

cache.expire(['myKey*', 'anotherKey'])

Asterisk * is used for wildcards

.addDisposer(callback)

Add callback to be called when an item is evicted by LRU-cache. Used to do cleanup

const cb = (key, value) => cleanup(key, value)
cache.addDisposer(cb)

.removeDisposer(callback)

Remove callback attached to LRU-cache

cache.removeDisposer(cb)

.debug([extraData])

Prints debug information about current cache (ie. hot keys, stale keys, keys in waiting state etc). Use extraData to add custom properties to the debug info, eg. hostname.

cache.debug({hostname: os.hostname()})

Examples

Note! These examples are written using ES2015 syntax. The lib is exported using Babel as CJS modules

Basic usage

import inMemoryCache from '@nrk/nodecache-as-promised'
const cache = inMemoryCache({ /* options */})

// imiplicit set cache on miss, or use cached value
cache.get('key', { worker: () => Promise.resolve({hello: 'world'}) })
  .then((data) => {
    console.log(data)
    // {
    //   value: {
    //     hello: 'world'
    //   },
    //   created: 123456789,
    //   cache: 'miss',
    //   TTL: 86400000
    // }
  })

Basic usage with options

import inMemoryCache from '@nrk/nodecache-as-promised';

const cache = inMemoryCache({
  initial: {                    // initial state
    foo: 'bar'
  },                            
  maxLength: 1000,              // LRU max object count
  maxAge: 24 * 60 * 60 * 1000   // LRU max age in ms
})
// set/overwrite cache key
cache.set('key', {hello: 'world'})
// imiplicit set cache on miss, or use cached value
cache.get('anotherkey', {
  worker: () => Promise.resolve({hello: 'world'}),
  ttl: 60 * 1000,               // TTL for cached object, in ms
  workerTimeout: 5 * 1000,      // worker timeout, in ms
  deltaWait: 5 * 1000,          // wait time, if worker fails
}).then((data) => {
    console.log(data)
    // {
    //   value: {
    //     hello: 'world'
    //   },
    //   created: 123456789,
    //   cache: 'miss',
    //   TTL: 86400000
    // }
  })

Distributed capabilites

Distributed expire and persisting of cache misses to Redis are provided as middlewares, ie. wrapping the in-memory cache with a factory that intercepts function calls. It should therefore be easy to write your own middlewares using pub/sub from rabbitMQ, zeroMQ, persisting to a NAS, hit/miss-ratio to external measurments systems and more.

Distributed expire

import inMemoryCache, {distCache} from '@nrk/nodecache-as-promised'
import Redis from 'ioredis'

// a factory function that returns a redisClient
const redisFactory = () => new Redis(/* options */)
const cache = inMemoryCache({initial: {fooKey: 'bar'}})
cache.use(distCache(redisFactory, 'namespace'))
// publish to redis (using wildcard)
cache.expire(['foo*'])
setTimeout(() => {
  cache.get('fooKey').then(console.log)
  // expired in server # 1 + 2
  // {value: {fooKey: 'bar'}, created: 123456789, cache: 'stale', TTL: 86400000}
}, 1000)

Persisting cache misses

import inMemoryCache, {persistentCache} from '@nrk/nodecache-as-promised'
import Redis from 'ioredis'

const redisFactory = () => new Redis(/* options */)
const cache = inMemoryCache({/* options */})
cache.use(persistentCache(
  redisFactory,
  {
    keySpace: 'myCache',   // key prefix used when storing in redis
    grace: 60 * 60         // auto expire unused keys in Redis after TTL + grace seconds
  }
))

cache.get('key', { worker: () => Promise.resolve('hello') })
// will store a key in redis, using key: myCache-<key>
// {value: 'hello', created: 123456789, cache: 'hit', TTL: 60000}

Persisting cache misses and distributed expire

import inMemoryCache, {distCache, persistentCache} from '@nrk/nodecache-as-promised'
import Redis from 'ioredis'

const redisFactory = () => new Redis(/* options */)
const cache = inMemoryCache({/* options */})
cache.use(distCache(redisFactory, 'namespace'))
cache.use(persistentCache(
  redisFactory,
  {
    keySpace: 'myCache',   // key prefix used when storing in redis
    grace: 60 * 60         // auto expire unused keys in Redis after TTL + grace seconds
  }
))

cache.expire(['foo*'])  // distributed expire of all keys starting with foo
cache.get('key', {
  worker: () => Promise.resolve('hello'),
  ttl: 60000,                       // in ms
  workerTimeout: 5000,
  deltaWait: 5000
}).then(console.log)
// will store a key in redis, using key: myCache-<key>
// {value: 'hello', created: 123456789, cache: 'miss', TTL: 60000}

---

Local development

First clone the repo and install its dependencies:

git clone [e-mail]:nrkno/nodecache-as-promised.git
git checkout -b feature/my-changes
cd nodecache-as-promised
npm install && npm run build && npm run test

Creating your own middleware

A middleware consists of three parts:

1. an exported factory function
2. constructor arguments to be used within the middleware
3. an exported facade that corresponds with the overriden functions (appending a next parameter that runs the next function in the middleware chain)

Lets say you want to build a middleware that notifies some other part of your application that a new value has been set (eg. using RxJs streams).

Here's an example on how to achieve this:

// export namespace to be applied in inMemoryCache.use().
export const streamingMiddleware = (onSet, onDispose) => (cacheInstance) => {
  // create a function that runs before the others in the middleware chain
  const set = (key, value, next) => {
    onSet(key, value)
    next(key, value)
  }

  // use functionality exposed by the inMemoryCache instance
  cacheInstance.addDisposer(onDispose)

  // export facade
  return {
    set
  }
}

Building and committing

After having applied changes, remember to build and run/fix tests before pushing the changes upstream.

# run the tests, generate code coverage report
npm run test
# inspect code coverage
open ./coverage/lcov-report/index.html
# update the code
npm run build
git commit -am "Add my changes"
git push origin feature/my-changes
# then make a PR to the master branch,
# and assign one of the maintainers to review your code

NOTE! Please make sure to keep commits small and clean (that the commit message actually refers to the updated files). Stylistically, make sure the commit message is Capitalized and starts with a verb in the present tense (eg. Add minification support).

License

MIT © NRK