txstate-utils

txstate-utils

Lightweight utility functions that can be used in a browser or in node.

Cache

Why use this?

Typical caches make users wait when a cache entry has expired, which could be a very long time for some cached processes, and could be a lot of users all waiting. This leads to consistent spikes of bad response times every X seconds or minutes if you graph your response times.

One common solution to this problem is to fill your cache automatically on an interval. However, this requires extra code and does extra work since it runs on a schedule instead of on-demand.

This cache works on-demand by refreshing entries in the background while immediately returning a slightly expired ("stale") result. If the resource is requested frequently, users will never have to wait and the spikes on your graph will vanish.

Additionally, typical caches that store objects lead most often to simple logic like

let obj = await cache.search(key)
if (typeof obj === 'undefined') obj = await goGetActualValue(key)

This is easy enough, but when you think about concurrent access, you realize that when a cache entry expires, any requests that come in before the cache is refilled (maybe it takes several seconds) will all call goGetActualValue instead of just the first one after expiration. This Cache class properly coalesces requests to prevent this issue, in addition to removing the need to write out that cache miss/cache hit boilerplate.

Overview

Cache entries go through multiple states after being fetched: fresh, stale, and expired. A fresh entry can be returned immediately. A stale entry can be returned immediately, but a new value should be fetched so that the next request might be fresh. An expired entry is not useful and the user must wait for a new value to be fetched.

Note that setting freshseconds === staleseconds renders this into a standard on-demand cache.

Example Usage

import { Cache } from 'txstate-utils'
const userCache = new Cache(id => User.findById(id))

function getUserWithCaching (id) {
  return userCache.get(id)
}
async function saveUser (userObj) {
  await userObj.save()
  await userCache.invalidate(userObj.id) // invalidate cache after update
  await userCache.refresh(userObj.id) // or go ahead and get it refreshed immediately
  // if you are confident userObj is the correct object to store in cache, you may set it
  // directly (a little risky as it avoids your fetching function and any logic it may be applying)
  await userCache.set(userObj.id, userObj)
}

Options

{
  freshseconds: period cache entry is fresh (default 5 minutes)
  staleseconds: period cache entry is stale (default 10 minutes)
  storageClass: an instance of a class that adheres to the storage engine
    interface (default is a simple in-memory cache)
  onRefresh: a callback that will be called any time a cache value is updated
    you could use this to implement a synchronization scheme between workers or instances
    any errors will be caught and logged without disrupting requests; if you have a custom
    logging scheme that does not use console.error, you should catch errors yourself
}

This is the storage engine interface:

interface StorageEngine {
  get (keystr:string): Promise<any>
  set (keystr:string, data:any): Promise<void>
  del (keystr:string): Promise<void>
  clear (): Promise<void>
}

storageClass will also accept an instance of lru-cache or memcached

If you wrap/implement your own storage engine, be aware that it is responsible for cleaning up expired cache entries or otherwise reducing its footprint. The default simple storage engine keeps everything in memory until expiration (it will efficiently garbage collect expired entries). LRU Cache and Memcache both have customizable storage limits. You're free to delete cache entries as aggressively as you see fit; a cache miss will not be harmful other than making the user wait for a result.

Advanced Usage

Compound Keys

Keys are automatically stringified with a stable JSON stringify, so you can use any JSON object structure as your cache key.

const personCache = new Cache(async ({ lastname, firstname }) => {
  return await getPersonByName(lastname, firstname)
})
const person = await personCache.get({ lastname: 'Smith', firstname: 'John' })

Note that you cannot use extra parameters on get and your fetcher function, as the second parameter is reserved for fetch helpers (see below).

Fetch Helpers

If your fetcher function requires some sort of context-sensitive helper to do its work (e.g. a request-scoped service), you may pass it in as a second parameter without affecting the lookup key:

const myCache = new Cache(async (key, service) => {
  return await service.findByKey(key)
})
const result = await myCache.get(key, service)

Tuning Guidance

Tuning freshseconds and staleseconds properly requires a little bit of data or intuition about usage. Here is some guidance to help you navigate:

• freshseconds
  • should be long enough that at least a few requests on average would come in during this interval
  • should not be so long that the data would be significantly less useful by the end
    • keep in mind that staleseconds should be even longer
  • does not need to be longer than the average time to complete the task
    • a very short freshseconds can increase the quality of your cached data while still removing most of the workload new cpu usage = uncached cpu usage / (requests per second * freshseconds)
• staleseconds
  • should be short enough that the data still holds most of its value by the end of the period
  • should be short enough that you don't run out of memory for the cache
    • alternatively, use LRU or memcache to control memory usage
  • staleseconds - freshseconds should be longer than average time between requests + average task completion time
    • otherwise users will wait for responses and spikes will appear on your response time graph

Alternate storage mechanisms

The default is to store cached results in memory in the node process until they expire, and then free them for garbage collection after expiration. This works well in most cases and is the fastest option. There are other options for you though:

• Memcache

  If you have multiple instances of your application, the default storage mechansism will store a full cache copy in each instance. This can lead to different instances giving different answers at the same moment. If that would cause problems for your application, adding memcached to your ecosystem could help. Once you have memcached configured and running, use the client library from npm memcached or memcache-client. Simply import and construct your client instance and then pass it into the Cache constructor as the storageClass option.

• LRU-Cache

  If you're fine with one full cache copy per instance, but you're afraid of the memory usage getting out of hand, you can use LRU-Cache to set a maximum number of cache entries (or use a custom sizeCalculation if some entries are much larger than others). Similar to memcache, just import and prepare an LRU cache instance and pass it in as the storageClass option.

  Note: if you use sizeCalculation, the object you're measuring will actually be stored underneath a data property, so do it like new LRUCache({ maxSize: 200, sizeCalculation: ({ data }) => data.length })

• Any other JSON storage mechanism

  Give the storageClass option an instance of a class that implements either StorageEngine<T> (for an asynchronous store like a database or redis) or SyncStorageEngine<T> (for an in-memory store that doesn't need promises). You just need to fill out the get, set, del and clear methods to wrap your store. Note that whatever storage mechanism you choose is responsible for its own cleanup after expiration. We do NOT call del when items expire, that's the storage mechanism's responsibility. We only call del when the user triggers an action that invalidates a record.