cascade-config

cascade-config

Asynchronous hierarchical config for node.js (env, argv, files, dirs) with inline var substitution and type conversion

Quick Start

cascade-config works by loading config objects from different sources and merging them together. 7 types of sources are provided:

• JS file: object is loaded using import-fresh
• directory: a full hierarchy of js files are loaded, reflecting the hierarchy in the loaded object
• env: object is composed from env vars
• envfile: object is loaded from a env file, using dotenv
• obj: object is explicitly specified
• args: object is composed from command line args
• yaml: YAML files, loaded with js-yaml

External loaders also exist as separated packages (see below)

The objects are loaded in the order their methods are called, so latter calls take precedence (an object loaded later would overwrite what is already loaded, by merge)

Also, one can specify as many loaders as required, even repeating types (that is, loading from more than one file is perfectly doable)

Finally, each object can be mounted in a specified path inside the final object, instead of its root

Let us see a quick example

const CC = require ('cascade-config');

const cconf = new CC();
const defaults = {...};

cconf
  .obj (defaults)
  .file (__dirname + '/etc/config.js',       {ignore_missing: true})
  .file (__dirname + '/etc/config-{env}.js', {ignore_missing: true})
  .env ({prefix: 'MYAPP_'})
  .args ()
  .done ((err, config) => {
    // merged config at 'config' object
  });

Variable substitution

CC supports using variables already read when calling certain loaders (js file, yaml and directory, for example). A very useful example is to use already-loaded config to specify the path of a file to load:

cconf
  .obj ({a:{b:'one'}})
  .file (__dirname + '/resources-{a.b}.js')
  .yaml (__dirname + '/other-resources-{a.b}.yaml')
  .done ((err, config) => {
    // config will contain what's in ./resources-one.js and in ./other-resources-one.yaml
  });

Variable substitution is made with string-interpolation so you can use any modifier allowed by it (defaults, transformations...)

Also, variable 'env' is always available, containing NODE_ENV. This makes it very simple to load configuration depending on the environment (production, development...):

cconf
  .file (__dirname + '/etc/config.js')
  .file (__dirname + '/etc/config-{env}.js')
  .done ((err, config) => {
    // if NODE_ENV=='development', it will load ./etc/config.js and
    // then ./etc/config-development.js
  });

Variable substitution works also on the loaded objects on each source: what is loaded so far in previous sources is used as source to substitute. Note that the substitution is applied only to the values (not to keys) and only if the value is a string. See an example:

// ENV is: APP_A=qwerty, APP_B__C__D=66
// cl is node index.js -a 66 --some.var=option_{B.C.D} --some__other__var=qwerty
// etc/config.js contains {z: {y: 'one_{some.var}_cc'}}
cconf
  .env ({prefix: 'APP_'})
    // env vars are available to substitute args...
  .args()
    // env vars and args are available to substitute file contents...
  .file (__dirname + '/etc/config.js')
  .done ((err, config) => {
    /*
    config would be
    {
      A: 'qwerty',
      B: {
        C: {
          D: 66
        }
      }
      a: 66,
      some: {
        var: 'option_66'
        other: {
          var: 'qwerty'
        }
      },
      z: {
        y: 'one_option_66_cc'
      }
    }
    */
  });

Notice z.y is built through 2 substitutions: some.var is built using B.C.D, and then z.y is built usint some.var

This works on all source types: if you want to provide a string verbatim, you can use the #str: type conversion, which will also prevent the variable substitution in it:

cconf
  .obj ({
    p1: '#str:some mustache {{a}} and other exotics: []%&_-|@',
  })
  .done ((err, config) => {
    /*
    config would be
    {
      p1: 'some mustache {{a}} and other exotics: []%&_-|@'
    }
    */
  });

Type conversion

Since variable substitution works only for string values, it is useful to have some sort of type conversion mechanism to convert string values into other types. cascade-config does this by looking whether the string begins with a specific prefix:

• '#int:' converts the rest of the string into an int (using parseInt)
• '#float:' converts the rest of the string into a float (using parseFloat)
• '#bool:' converts the rest of the string into a boolean (as in value === 'true')
• '#base64:' converts the rest of the string into a Buffer by base64-decoding it
• '#str:': passes the string verbatim without any variable substitution. Useful if the value contains curly braces in its own
• '#csv:': splits the string by commas, trims each element and returns the resulting array
• '#json:': converts the string to an object, parsing it as json. If the string is not valid json, returns the original string

Type conversions can be applied to any string, not just those containing a variable substitution: it just need to be at the beginnign of the string. This is very useful to pass complex (non-string) config via cli or env vars

Let us see an example:

cconf
.obj ({a: 1, b: '2', c: 'true', d: 'SmF2YVNjcmlwdA==', e: 67.89, f:'123.456'})
.obj ({
  p1: '#int:{a}',
  p2: '#int:{b}',
  p3: '#int:{c}',
  p4: '#bool:{c}',
  p5: '#base64:{d}',
  p6: '#float:{e}',
  p7: '#float:{f}'
}).
obj ({
  verbatim: {
    a: '#int:1234',
    b: '#bool:true',
    c: '#float:12.344e-3',
    d: '#base64:cXdlcnR5dWlvcAo=',
    e: '#csv: aaa, fff , ggg',
    f: '#str:{a} {{f}}',
    g: '#json:{"aa":5, "bb":"qaz"}'
  },
  ill_converts: {
    a: '#int:aa',
    b: '#bool:null',
    c: '#float:____',
    d: '#json:{"aa":5, "bb:"qaz"}'
  }
})
.done ((err, config) => {
/*
config would be
{
  a: 1,
  b: '2',
  c: 'true',
  d: 'SmF2YVNjcmlwdA==',
  e: 67.89,
  f: '123.456',
  p1: 1,
  p2: 2,
  p3: NaN,
  p4: true,
  p5: <Buffer 4a 61 76 61 53 63 72 69 70 74>,
  p6: 67.89,
  p7: 123.456,
  verbatim: {
    a: 1234,
    b: true,
    c: 0.012344,
    d: <Buffer 71 77 65 72 74 79 75 69 6f 70 0a>,
    e: [ 'aaa', 'fff', 'ggg' ],
    f: '{a} {{f}}',
    g: { aa: 5, bb: 'qaz' }
  },
  ill_converts: { a: NaN, b: false, c: NaN, d: '{"aa":5, "bb:"qaz"}' }
}

*/
});

Note that dotenv starting with version 15.0.0 treats # as start of comment, unless the value is wrapped in double quotes; therefore to use this feature on an envfile you will need to elcose the value in double quotes:

a__b__c="#int:{previous_def_1}"
d__e = "#int:{previous_def_2}"

Value loaders

Similar to the type conversions, which apply transformations to string values, there are similar goodies to provide extra loading capabilities:

• #file:: takes the rest of the string, opens it as a file and substitutes the whole value with its contents (also a string)
• #jsfile:: takes the rest of the string, opens it as a js object and substitutes the whole value with its contents (an object)
• #yamlfile:: takes the rest of the string, opens it as a js object and substitutes the whole value with its contents (an object)

In the case of #jsfile and #yamlfile the loaded object is subject to further variable expansions, type conversions and value loaders

Examples:

Text file val.txt:

this is a test file
this is a test file
this is a test file

{
  a: '#file:./val.txt',
  b: 6
}

becomes

{
  a: 'this is a test file
this is a test file
this is a test file',
  b: 6
}

YAML file val.yaml:

z:
 - 1
 - 2

{
  a: '#yamlfile:./val.yaml',
  b: 6
}

becomes

{
  a: {
    z: [1, 2]
  }
  b: 6
}

API

• .obj(object, opts): loads and merges an object, verbatim. Useful to provide defaults (if loaded first) or overrides (if loaded last) Available options on opts:

  • mount: where to merge the loaded object inside the main result. The path uses the same semantics than lodash's _.set(obj, path, ...)

• .env(opts): loads and merges an object composed with env vars. opts can be passed to control what env vars to pick:

  • mount: where to merge the loaded object inside the main result. The path uses the same semantics than lodash's _.set(obj, path, ...)
  • prefix: str: selects all vars with name starting with str, and removes the prefix before adding it to the object
  • regexp: regex: selects all vars whose name matches regex

  In all cases, one can produce deep objects (ie subobjects) by adding __ to the var name: it will be treated as a .

  // ENV is: APP_A=qwerty, APP_B__C__D=66, SOME__OTHER__VAR=0, AND__ANOTHER__VAR=8
  cconf
    .env ({prefix: 'APP_'})
    .env ({regexp: /OTHER/})
    .done ((err, config) => {
      /* config would be
      {
        A: 'querty',
        B: {
          C: {
            D: 66
          }
        },
        SOME: {
          OTHER: {
            VAR: 0
          }
        }
      }
      */
    });
  

• .args(opts): loads and merges an object composed with the command line args passed (parsed by minimist). As in the case of env() all occurrences of __ are converted to ., so one can use either to specify hierarchy

  // cl is node index.js -a 66 --some.var=rt --some__other__var=qwerty
  cconf.args().done ((err, config) => {
    /*
    config would be
    {
      a: 66,
      some: {
        var: 'rt',
        other: {
          var: 'qwerty'
        }
      }
    }
    */
  });
  
  cconf.args({prefix: 'some.'}).done ((err, config) => {
    /*
    config would be
    {
      var: 'rt',
      other: {
        var: 'qwerty'
      }
    }
    */
  });
  

  Allowed options are:

  • mount: where to merge the loaded object inside the main result. The path uses the same semantics than lodash's _.set(obj, path, ...)
  • input: a string that would be used as source for minimist instead of process.argv.slice(2)
  • prefix: str: selects all vars with name starting with str, and removes the prefix before adding it to the object
  • regexp: regex: selects all vars whose name matches regex

• .file(filename, opts): loads object from a javascript file. filename supports variable substitution. Options are:

  • mount: where to merge the loaded object inside the main result. The path uses the same semantics than lodash's _.set(obj, path, ...)
  • ignore_missing: if truish, just return an empty object if the file can not be read; if false, raise an error. Defaults to false

• .envfile(filename, opts): loads object from an envfile. filename supports variable substitution. Options are:

  • mount: where to merge the loaded object inside the main result. The path uses the same semantics than lodash's _.set(obj, path, ...)
  • ignore_missing: if truish, just return an empty object if the file can not be read; if false, raise an error. Defaults to false
  • prefix: str: selects all vars with name starting with str, and removes the prefix before adding it to the object
  • regexp: regex: selects all vars whose name matches regex

• .directory(opts): loads a single object composed by an entire file hierarchy. Only js and json files are considered, and the resulting object reflects the relative path of the file. That is, a file a/b/c.js containing {n:1, b:6} would produce {a: {b: {c: {n: 1, b: 6}}}}. Also, dots in file or dir names are changed into _. Options are:

  • mount: where to merge the loaded object inside the main result. The path uses the same semantics than lodash's _.set(obj, path, ...)
  • files: base dir to read files from. defaults to __dirname + '/etc', and supports variable substitution

• .yaml(filename, opts): loads object from a YAML file. filename supports variable substitution. Options are:

  • mount: where to merge the loaded object inside the main result. The path uses the same semantics than lodash's _.set(obj, path, ...)
  • ignore_missing: if truish, just return an empty object if the file can not be read; if false, raise an error. Defaults to false

Extended API

The api exposed so far provides a simple, plain JS object with all the config; this is usually more than enough, but for more complex use cases -where advanced config management is needed- a more powerful interface is provided

This extender interface is selected by simply passing {extended: true} as second param of .done():

  cconf
    .args({prefix: 'some.'})
    ...
    .done ((err, config) => {
     ...
    }, {extended: true}
  );

In this case config is no longer a plain object containing the config, but an interface to it with the following methods:

• config(): returns the plain config object (as in the standard interface)
• get(): gets a value or slice from the config. Uses the same interface, and has the same logic than lodash's _.get(obj, ...)
• set(): sets a value or slice in the config. Uses the same interface, and has the same logic than lodash's _.set(obj, ...)
• unset(): unsets a value or slice in the config. Uses the same interface, and has the same logic than lodash's _.unset(obj, ...)
• reload (cb): rereads all config again, as if you called done(). It has, in fact, the same interface
• onChange(fn): registers a function to be called every the the config is changed (by calling set(), unset() or reload()). Teh function will be called every time the config is changed, with the following params:
  • function (path): where path is the path of the change within the configuration, or null if unknown or affects all the config

Note: the object returned by config() is mutable, but the object reference itself does not change: if you save it for later, you can read the new config in it after any change, reload() included, as expected

External loaders

There are external packages that add loaders to cascade-config, thus allowing to read config from other type of sources:

• cascade-config-mongodb : reads config from mongodb databases
• cascade-config-http : reads config from http URLs (with JSON payloads)