fp-tree-and-member-cache

FP-Tree-And-Member-Cache

Wrapper class for redis-value-cache adapted for Trees and Members.
Provides Search for kind with name, id or shortName for trees.
Provides Search for members by id, group, location, position or link.
On updates over Redis pub/sub the values are dropped from the cache. Uses the etag property to decide if there were changes.
TreeAndMemberCache needs to be connected before it can be used.

Options

Options for creating a treeAndMemberCache.

• redis: Options for connection to Redis.
  • redis.clientOpts: Options for the node-redis client (details). Used for both subscriber and client.
  • redis.client (Optional): Redis client to duplicate for both subscriber and client. Will be prioritized over redis.clientOpts if both are provided.
  • channel: name of the channel for updates.
• fallback (Optional):
  • fallbackFetchMethod (Optional): Function returning an array of Trees or Members and optionally the etag for this directory.
  • fallbackOptions (Optional/if both are supplied fallbackFetchMethod wil be prioritized):
    • url: url of the endpoint the members and trees can be fetched from (e.g. url/scope/dscId/directoryId)
    • userAgent (Optional): value to set for the "User-Agent" header property
• cacheFallbackValues (Optional): Whether to cache Trees/Members from the fallbackFetchMethod (Default: false).
• cacheMaxSize (Optional): Maximum amount of objects to store in cache (Default 500).
• trackStatistics (Optional): Whether the tmc should track statistic data

Usage (Typescript)

import {TreeAndMemberCache} from "fp-tree-and-member-cache"
import type { Options } from "fp-tree-and-member-cache";

type Tree = {
	id: number;
	children: Tree[];
	data: {
		shortName?: string;
	};
	name: string;
	kind: "POSITION" | "SECURITY" | "LOCATION" | "GROUP";
}

type Member = {
	linktype: string;
	linkid: string;
	pos: number;
	grp: number;
	loc: number;
	id: string;
}

const opts: Options<Tree, Member> = {
	redis: {
		clientOpts: {
			socket: {
				port: 6379,
				host: "localhost"
			},
			name: "..."
		},
		channel: "channel_name"
	},
	fallback: {
		fallbackFetchMethod: async (scope: "trees" | "members", dscId: number, directoryId: number) => {
			// simplified example
			const resp = await fetch("...", {headers: {"User-Agent": "..." }})

			if (!resp.ok) {
				return null;
			}

			const etag = resp.headers.get("etag");

			if (scope === "trees") {
				const treeBody = await resp.json() as {tree: Tree[]};

				return {
					values: treeBody.tree,
					etag: etag ?? undefined
				}
			} 
			const memberBody = await resp.json() as {rows: Member[]};

			return {
				values: memberBody.rows,
				etag: etag ?? undefined
			}
		}
	},
	cacheFallbackValues: true,
	cacheMaxSize: 250,
	trackStatistics: true
}

// Either connect tmc after creation ...

const tmc1 = new TreeAndMemberCache(opts);

await tmc1.connect();

const result = await tmc1.getTreeNodeByKindAndId(10, 100, "SECURITY", 13379);
	
console.log(result?.treeNode);

await tmc1.disconnect();
// ... or create already connected tmc

const opts2: Options<Tree, Member> = {
	redis: {
		clientOpts: {
			socket: {
				port: 6379,
				host: "localhost"
			},
			name: "..."
		},
		channel: "channel_name"
	},
	fallback: {
		fallbackOptions: {
			url: "...",
			userAgent: "..."
		}
	},
	cacheFallbackValues: true,
	cacheMaxSize: 250,
	trackStatistics: true
}

const tmc2 = await TreeAndMemberCache.new(opts2);

const result2 = await tmc1.getTreeNodeByKindAndId(10, 100, "SECURITY", 13379);
	
console.log(result2?.treeNode);

await tmc2.disconnect();

Emitted Events

Name	When	Listener arguments
ready	RedisValueCache is ready to use	No arguments
connection-error	Either the client or the subscriber emitted an error event	(error: Error, client: "subscriber" | "client")
dropped	A value was dropped because of a message	(scope: "tree" | "member" | "directory", dscId: number, directoryId: number,etag: string | undefined)

If an connection-error event is emitted the TreeAndMemberCache flushes the cache because it could miss messages.

Removing values from cache

If you do not use the delete function the only way values are removed from the cache are via messages in the redis channel. Additionally to receiving a message abut a directory the etag of the message needs to be different from the saved etag to remove the directory.

fallbackFetchMethod

The fallbackFetchMethod is used if a value can not be found in redis. It is important that the array of trees returned by this function only includes roots.

You do not have to return an etag in the fallbackFetchMethod. This would mean the directory will be removed from the cache on any update message for the directory.

It is recommended to set cacheFallbackValues to true if you are using the fallbackFetchMethode, because the additional effort put into indexing the trees/members would be wasted otherwise. If you do not want to cache fallback values it might be better to check if the cache returned a value and then fetch the directory yourself.

Statistics

The statistics are a simple measure of how long the functions of the tmc take and how often they are called.
They look like this:

interface ReturnedStatistics {
	// how often the function was called an there was a value cached
	callsCached: number;
	// how often the function was called an there was no value cached
	callsMiss: number;
	// the average time it took to complete the function when a value was cached
	averageTimeCached: number;
	// the average time it took to complete the function when no value was cached
	averageTimeMiss: number;
	// the max time it took to complete the function when a value was cached
	maxTimeCached: number;
	// the max time it took to complete the function when no value was cached
	maxTimeMiss: number;
	// how often a value was found when bo value was cached
	valueFoundMiss: number;
	// the name of the function
	functionName: string;
}

The collected times are in ms rounded to the 3rd decimal place.
The included functions are all functions that get a Tree or Member object excluding the "WithoutChildren" functions for trees, since they call the normal function which is counted.

const statistics1: ReturnedStatistics[] = tmc.getFunctionStatistics();

const statistics2: {directoryId: number, dscId:number, statistics: ReturnedStatistics[]}[] = tmc.getFunctionStatisticsBasedOnDscIdAndDirectory();

console.log(statistics1);

console.log(statistics2);

If the trackStatistics option is not set both arrays will be empty.