@travetto/base

Base

Environment config and common utilities for travetto applications.

Install: @travetto/base

npm install @travetto/base

# or

yarn add @travetto/base

Base is the foundation of all Travetto applications. It is intended to be a minimal application set, as well as support for commonly shared functionality. It has support for the following key areas:

• Environment Support
• Runtime Flags
• Console Management
• Resource Access
• Standard Error Support
• Stream Utilities
• Object Utilities
• Data Utilities
• Common Utilities
• Time Utilities
• Process Execution
• Shutdown Management

Environment Support

The functionality we support for testing and retrieving environment information for known environment variables. They can be accessed directly on the Env object, and will return a scoped EnvProp, that is compatible with the property definition. E.g. only showing boolean related fields when the underlying flag supports true or false

Code: Base Known Environment Flags

interface TravettoEnv {
    /** 
     * The node environment we are running in
     * @default development
     */
    NODE_ENV: 'development' | 'production';
    /** 
     * Outputs all console.debug messages, defaults to `local` in dev, and `off` in prod. 
     */
    DEBUG: boolean | string;
    /** 
     * Environment to deploy, defaults to `NODE_ENV` if not `TRV_ENV` is not specified.  
     */
    TRV_ENV: string;
    /** 
     * Special role to run as, used to access additional files from the manifest during runtime.  
     */
    TRV_ROLE: Role;
    /** 
     * Whether or not to run the program in dynamic mode, allowing for real-time updates  
     */
    TRV_DYNAMIC: boolean;
    /** 
     * The folders to use for resource lookup
     */
    TRV_RESOURCES: string[];
    /** 
     * The max time to wait for shutdown to finish after initial SIGINT, 
     * @default 2s
     */
    TRV_SHUTDOWN_WAIT: TimeSpan | number;
    /**
     * The desired runtime module 
     */
    TRV_MODULE: string;
    /**
     * The location of the manifest file
     * @default undefined
     */
    TRV_MANIFEST: string;
    /**
     * trvc log level
     */
    TRV_BUILD: 'none' | 'info' | 'debug' | 'error' | 'warn',
  }

Environment Property

For a given EnvProp, we support the ability to access different properties as a means to better facilitate environment variable usage.

Code: EnvProp Shape

export class EnvProp<T> {
  constructor(public readonly key: string) { }
  /** Remove value */
  clear(): void;
  /** Export value */
  export(val: T | undefined): Record<string, string>;
  /** Read value as string */
  get val(): string | undefined;
  /** Read value as list */
  get list(): string[] | undefined;
  /** Add values to list */
  add(...items: string[]): void;
  /** Read value as int  */
  get int(): number | undefined;
  /** Read value as boolean */
  get bool(): boolean | undefined;
  /** Read value as a time value */
  get time(): number | undefined;
  /** Determine if the underlying value is truthy */
  get isTrue(): boolean;
  /** Determine if the underlying value is falsy */
  get isFalse(): boolean;
  /** Determine if the underlying value is set */
  get isSet(): boolean;
}

Runtime Flags

Env also provides some convenience methods for common flags used at runtime within the framework. These are wrappers around direct access to process.env values with a little bit of logic sprinkled in.

Code: Provided Flags

export const Env = delegate({
  /** Get name */
  get name(): string | undefined {
    return process.env.TRV_ENV || (!prod() ? 'local' : undefined);
  },

  /** Are we in development mode */
  get production(): boolean {
    return prod();
  },

  /** Is the app in dynamic mode? */
  get dynamic(): boolean {
    return IS_TRUE.test(process.env.TRV_DYNAMIC!);
  },

  /** Get debug value */
  get debug(): false | string {
    const val = process.env.DEBUG ?? '';
    return (!val && prod()) || IS_FALSE.test(val) ? false : val;
  }
});

Resource Access

The primary access patterns for resources, is to directly request a file, and to resolve that file either via file-system look up or leveraging the Manifest's data for what resources were found at manifesting time.

The FileLoader allows for accessing information about the resources, and subsequently reading the file as text/binary or to access the resource as a Readable stream. If a file is not found, it will throw an AppError with a category of 'notfound'.

The ResourceLoader extends FileLoader and utilizes the Env's TRV_RESOURCES information on where to attempt to find a requested resource.

Standard Error Support

While the framework is 100 % compatible with standard Error instances, there are cases in which additional functionality is desired. Within the framework we use AppError (or its derivatives) to represent framework errors. This class is available for use in your own projects. Some of the additional benefits of using this class is enhanced error reporting, as well as better integration with other modules (e.g. the RESTful API module and HTTP status codes).

The AppError takes in a message, and an optional payload and / or error classification. The currently supported error classifications are:

• general - General purpose errors
• system - Synonym for general
• data - Data format, content, etc are incorrect. Generally correlated to bad input.
• permission - Operation failed due to lack of permissions
• auth - Operation failed due to lack of authentication
• missing - Resource was not found when requested
• timeout - Operation did not finish in a timely manner
• unavailable - Resource was unresponsive

Console Management

This module provides logging functionality, built upon console operations.

The supported operations are:

• console.error which logs at the ERROR level
• console.warn which logs at the WARN level
• console.info which logs at the INFO level
• console.debug which logs at the DEBUG level
• console.log which logs at the INFO level

Note: All other console methods are excluded, specifically trace, inspect, dir, time/timeEnd

How Logging is Instrumented

All of the logging instrumentation occurs at transpilation time. All console.* methods are replaced with a call to a globally defined variable that delegates to the ConsoleManager. This module, hooks into the ConsoleManager and receives all logging events from all files compiled by the Travetto.

A sample of the instrumentation would be:

Code: Sample logging at various levels

export function work() {
  console.debug('Start Work');

  try {
    1 / 0;
  } catch (err) {
    console.error('Divide by zero', { error: err });
  }
  console.debug('End Work');
}

Code: Sample After Transpilation

"use strict";
Object.defineProperty(exports, "__esModule", { value: true });
exports.work = void 0;
const tslib_1 = require("tslib");
const ᚕ_c = tslib_1.__importStar(require("@travetto/base/src/console.js"));
var ᚕf = "@travetto/base/doc/transpile.js";
function work() {
    ᚕ_c.log({ level: "debug", source: ᚕf, line: 2, scope: "work", args: ['Start Work'] });
    try {
        1 / 0;
    }
    catch (err) {
        ᚕ_c.log({ level: "error", source: ᚕf, line: 7, scope: "work", args: ['Divide by zero', { error: err }] });
    }
    ᚕ_c.log({ level: "debug", source: ᚕf, line: 9, scope: "work", args: ['End Work'] });
}
exports.work = work;

Filtering Debug

The debug messages can be filtered using the patterns from the debug. You can specify wild cards to only DEBUG specific modules, folders or files. You can specify multiple, and you can also add negations to exclude specific packages.

Terminal: Sample environment flags

# Debug
$ DEBUG=-@travetto/model npx trv run app
$ DEBUG=-@travetto/registry npx trv run app
$ DEBUG=@travetto/rest npx trv run app
$ DEBUG=@travetto/*,-@travetto/model npx trv run app

Additionally, the logging framework will merge debug into the output stream, and supports the standard usage

Terminal: Sample environment flags for standard usage

# Debug
$ DEBUG=express:*,@travetto/rest npx trv run rest

Stream Utilities

The StreamUtil class provides basic stream utilities for use within the framework:

• toBuffer(src: Readable | Buffer | string): Promise<Buffer> for converting a stream/buffer/filepath to a Buffer.
• toReadable(src: Readable | Buffer | string):Promise<Readable> for converting a stream/buffer/filepath to a Readable
• writeToFile(src: Readable, out: string):Promise<void> will stream a readable into a file path, and wait for completion.
• waitForCompletion(src: Readable, finish:()=>Promise<any>) will ensure the stream remains open until the promise finish produces is satisfied.

Object Utilities

Simple functions for providing a minimal facsimile to lodash, but without all the weight. Currently ObjectUtil includes:

• isPrimitive(el) determines if el is a string, boolean, number or RegExp
• isPlainObject(obj) determines if the obj is a simple object
• isFunction(o) determines if o is a simple Function
• isClass(o) determines if o is a class constructor
• isSimple(a) determines if a is a simple value
• isPromise(a) determines if a is a promise

Data Utilities

Data utilities for binding values, and type conversion. Currently DataUtil includes:

• deepAssign(a, b, mode ?) which allows for deep assignment of b onto a, the mode determines how aggressive the assignment is, and how flexible it is. mode can have any of the following values:

  • loose, which is the default is the most lenient. It will not error out, and overwrites will always happen
  • coerce, will attempt to force values from b to fit the types of a, and if it can't it will error out
  • strict, will error out if the types do not match

• coerceType(input: unknown, type: Class<unknown>, strict = true) which allows for converting an input type into a specified type instance, or throw an error if the types are incompatible.

• shallowClone<T = unknown>(a: T): T will shallowly clone a field

• filterByKeys<T>(obj: T, exclude: (string | RegExp)[]): T will filter a given object, and return a plain object (if applicable) with fields excluded using the values in the exclude input

Common Utilities

Common utilities used throughout the framework. Currently Util includes:

• uuid(len: number) generates a simple uuid for use within the application.
• allowDenyMatcher(rules[]) builds a matching function that leverages the rules as an allow/deny list, where order of the rules matters. Negative rules are prefixed by '!'.
• naiveHash(text: string) produces a fast, and simplistic hash. No guarantees are made, but performs more than adequately for framework purposes.
• shortHash(text: string) produces a sha512 hash and returns the first 32 characters.
• fullHash(text: string, size?: number) produces a full sha512 hash.
• resolvablePromise() produces a Promise instance with the resolve and reject methods attached to the instance. This is extremely useful for integrating promises into async iterations, or any other situation in which the promise creation and the execution flow don't always match up.

Code: Sample makeTemplate Usage

const tpl = makeTemplate((name: 'age'|'name', val) => `**${name}: ${val}**`); 
tpl`{{age:20}} {{name: 'bob'}}</>;
// produces
'**age: 20** **name: bob**'

Time Utilities

TimeUtil contains general helper methods, created to assist with time-based inputs via environment variables, command line interfaces, and other string-heavy based input.

Code: Time Utilities

export class TimeUtil {
  /**
   * Test to see if a string is valid for relative time
   * @param val
   */
  static isTimeSpan(val: string): val is TimeSpan;
  /**
   * Returns time units convert to ms
   * @param amount Number of units to extend
   * @param unit Time unit to extend ('ms', 's', 'm', 'h', 'd', 'w', 'y')
   */
  static timeToMs(amount: number | TimeSpan, unit?: TimeUnit): number;
  /**
   * Resolve time or span to possible time
   */
  static resolveInput(value: number | string | undefined): number | undefined;
  /**
   * Returns a new date with `amount` units into the future
   * @param amount Number of units to extend
   * @param unit Time unit to extend ('ms', 's', 'm', 'h', 'd', 'w', 'y')
   */
  static timeFromNow(amount: number | TimeSpan, unit: TimeUnit = 'ms'): Date;
  /**
   * Pretty print a delta between now and `time`, with auto-detection of largest unit
   */
  static prettyDeltaSinceTime(time: number, unit?: TimeUnit): string;
  /**
   * Pretty print a delta, with auto-detection of largest unit
   * @param delta The number of milliseconds in the delta
   */
  static prettyDelta(delta: number, unit?: TimeUnit): string;
}

Process Execution

Just like child_process, the ExecUtil exposes spawn and fork. These are generally wrappers around the underlying functionality. In addition to the base functionality, each of those functions is converted to a Promise structure, that throws an error on an non-zero return status.

A simple example would be:

Code: Running a directory listing via ls

import { ExecUtil } from '@travetto/base';

export async function executeListing() {
  const { result } = ExecUtil.spawn('ls');
  const final = await result;
  console.log('Listing', { lines: final.stdout.split('\n') });
}

As you can see, the call returns not only the child process information, but the Promise to wait for. Additionally, some common patterns are provided for the default construction of the child process. In addition to the standard options for running child processes, the module allows for the following execution options:

Code: Execution Options

export interface ExecutionOptions extends SpawnOptions {
  /**
   * Should an error be caught and returned as a result. Determines whether an exit code > 0 throws
   * an Error, or if it merely marks the process as completed, marking the result as invalid.
   */
  catchAsResult?: boolean;
  /**
   * Should the environment be isolated, or inherit from process.env
   */
  isolatedEnv?: boolean;
  /**
   * Built in timeout for any execution. The number of milliseconds the process can run before
   * terminating and throwing an error
   */
  timeout?: number;
  /**
   * Determines how to treat the stdout/stderr data.
   *  - 'text' will assume the output streams are textual, and will convert to unicode data.
   *  - 'text-stream' makes the same assumptions as 'text', but will only fire events, and will
   *        not persist any data.  This is really useful for long running programs.
   *  - 'binary' treats all stdout/stderr data as raw buffers, and will not perform any transformations.
   *  - 'raw' avoids touching stdout/stderr altogether, and leaves it up to the caller to decide.
   */
  outputMode?: 'raw' | 'binary' | 'text' | 'text-stream';
  /**
   * On stderr line.  Requires 'outputMode' to be either 'text' or 'text-stream'
   */
  onStdErrorLine?: (line: string) => void;
  /**
   * On stdout line.  Requires 'outputMode' to be either 'text' or 'text-stream'
   */
  onStdOutLine?: (line: string) => void;
  /**
   * The stdin source for the execution
   */
  stdin?: string | Buffer | Readable;
}

Shutdown Management

Another key lifecycle is the process of shutting down. The framework provides centralized functionality for running operations on graceful shutdown. Primarily used by the framework for cleanup operations, this provides a clean interface for registering shutdown handlers. The code intercepts SIGTERM and SIGUSR2, with a default threshold of 2 seconds. These events will start the shutdown process, but also clear out the pending queue. If a kill signal is sent again, it will complete immediately.

As a registered shutdown handler, you can do.

Code: Registering a shutdown handler

import { ShutdownManager } from '@travetto/base';

export function registerShutdownHandler() {
  ShutdownManager.onGracefulShutdown(async () => {
    // Do important work, the framework will wait until all async
    //   operations are completed before finishing shutdown
  });
}