bigfig

bigfig

configuration objects that vary based on multiple considerations

Intended Audience: Applications which need to be configured in particular ways for different environments.

Features

customize configuration based on many concerns

• for some apps a single "development versus production" distinction might be too simple
• arbitrary set of names and values, which you define, to describe when the config should be customized
• library assumes no keys or values (not even based on NODE_ENV), so it doesn't dictate how you solve your problem
• the same source can be used to configure both the client (customized for each request) and the server (customized to the server environment), sharing details as appropriate and protecting details as appropriate

works on JSON-like objects

• scalars and arrays and objects
• arbitrarily deeply nested objects

focused on one problem

• allows you to use other libraries for their strengths
• can use other libraries for custom disk formats (yaml, cson, json5, etc)
• can use your own disk access algorithms (sync or async, cached reads, etc)

customizable

• can use your own object merge algorithm (for example, to adjust how arrays are merged)
• can use your own match algorithm (for example, to match a dynamic range of values)

optimized

• constructor studies source so that read() runs as fast a possible
• match and merge algorithms have been optimized for speed and low GC overhead

Example

This example is a bit complex to demonstrate some of the value/features of this library. Your config can be as simple/complex as you need it to be 😀

var bigfig = require("./index.js");
var fig, config;
fig = new bigfig.Config({
        // default (development)
        apiURL: "http://localhost:3001/",
        assetURL: "http://localhost:3000/static",

        // don't expose this config to the client!
        "__context?runtime=server": {
            listenPort: 3000,
            memcache: {
                host: "localhost",
                port: 11211
            },
        },

        // stand-alone staging environment for validation testing
        "__context?env=staging": {
            listenPort: 80,
            apiURL: "http://staging.mysite.com:4080/",
            assetURL: "http://staging.mysite.com/static",
            memcache: {
                host: "memcache.staging.mysite.com"
            }
        },

        // adjust the config for production hosts
        "__context?env=production": {
            apiURL: "http://api.mysite.com/",
            assetURL: "http://cdn.provider.com/mysite/",
            "__context?secure=true": {
                assetURL: "https://cdn.provider.com/mysite/"
            }
        },

        "__context?env=production&runtime=server": {
            listenPort: 80,
            // perhaps you have more than one type of production host
            "__context?colo=east": {
                apiURL: "http://api.east.mysite.com:4080/",
                memcache: {
                    host: "memcache.east.mysite.com",
                    // for some legacy reason the eastern memcached is on a weird port
                    port: 11666
                }
            },
            "__context?colo=west": {
                apiURL: "http:/api.west.mysite.com:4080/",
                memcache: {
                    host: "memcache.west.mysite.com"
                }
            }
        }
    });

config = fig.read({
    runtime: "server",
    env: "production",  // might come from process.env.NODE_ENV
    colo: "east"        // might be interpreted from the hostname
});
// {
//     apiURL: 'http://api.east.mysite.com:4080/',
//     assetURL: 'http://cdn.provider.com/mysite/',
//     listenPort: 80,
//     memcache: {
//         host: 'memcache.east.mysite.com',
//         port: 11666
//     }
// }

config = fig.read({
    runtime: "client",
    env: "production",  // might come from process.env.NODE_ENV
    secure: "true",     // might come from req.protocol === "https"
});
// {
//     apiURL: 'http://api.mysite.com/',
//     assetURL: 'https://cdn.provider.com/mysite/'
// }

Source Object Format

The source of the configuration is a JSON-like object -- an object with scalars, and objects and arrays which can be nested arbitrarily deep.

{
    port: 80,
    memcache: {
        host: "localhost",
        port: 11211,
        settings: {
            timeout: 1000
        }
    }
}

This simple config we call a default or root.

(If this simple approach meets your needs then you probably don't need this library 😀)

You can add sections, each of which describes how the config should be different for a different situation. The situation is described by a set of keys and values we call a context.

{
    port: 80,
    memcache: {
        host: "localhost",
        port: 11211,
        settings: {
            timeout: 1000
        }
    },
    "__context?env=staging": {
        memcache: {
            host: "memcache.staging.mysite.com"
        }
    },
    "__context?env=production": {
        memcache: {
            host: "memcache.mysite.com"
        }
    }
}

As well, a section can have further speciallizations within it.

{
    port: 80,
    memcache: {
        host: "localhost",
        port: 11211,
        settings: {
            timeout: 1000
        }
    },
    "__context?env=production": {
        "__context?colo=east": {
            memcache: {
                host: "memcache.east.mysite.com"
            }
        },
        "__context?colo=west": {
            memcache: {
                host: "memcache.west.mysite.com"
            }
        },
    }
}

The section specializations can occur arbitrarily deep.

{
    port: 80,
    memcache: {
        host: "localhost",
        port: 11211,
        settings: {
            timeout: 1000,
            "__context?env=production": {
                timeout: 500
            }
        }
    }
}

The context keys have these properties:

• start with __context?
• keys and values formatted the same as URL query parameters. for example __context?env=production&colo=east
• special characters should be encoded just as for URL query parameters (%xx)

FYI, the __context?... keys don't need to be quoted in YAML files.

API Reference

Config(source, options) constructor throws

This creates a new bigfig object, on which you can call read() multiple times with different contexts.

• source {Object} the source of the configuration, as describe in Source Object Format above
• options {Object} an optional object containing settings used to adjust how the source is interpretted

There currently are no defined options.

This constructor will intentionally throw an error on the following conditions:

• The source is not an object. (Arrays and scalars are not accepted.)
• The source has a subsection which redefines a context key. Example:

{
    color: 'red',
    "__context?env=production": {
        color: 'green',
        "__context?env=development": {
            color: 'blue',
        }
    }
}

Config.read(context, options) method

Creates a config object, customized to the specified context.

• context {Object} a simple object with dimension names and values
• options {Object} an optional object containing settings used to adjust how the config object is created

There currently are no defined options.

Config.match(context, options) method

This lower-level method isn't normally called. It returns all sections which match the context.

• context {Object} a simple object with dimension names and values
• options {Object} an optional object containing settings used to adjust how the sections are matched
• returns {Array} an array of config objects to merge

There currently are no defined options.

Config.merge(sections, options) method

This lower-level method isn't normally called. It merges the sections into a configuration object.

• sections {Array} an array of config objects to merge
• options {Object} an optional object containing settings used to adjust how the configs are merged
• returns {Object} a config object

There currently are no defined options.

matcher(sectionContext, runContext, options)

The default match algorithm. See Customizing the Match Algorithm below for details on how to replace this with your own algorithm.

• sectionContext {Object} the context generated from the __context? keys in the source
• runContext {Object} the context passed to read() or match()
• options {Object} an optional object containing settings used to adjust how the contexts are matched
• returns {Boolean} true if runContext matches sectionContext

There currently are no defined options.

cloner(oldObject)

This is a low-level utility for cloning an object. You usually don't need to call or overrride this function.

• oldObject {Object} the object to clone
• returns {Object} a copy of the object

merger(base, changes, options)

The default merge algorithm. See Customizing the Merge Algorithm below for details on how to replace this with your own algorithm.

• base {Object} object whose keys and values will be modified by changes
• changes {Object} object which contains changes to apply to base
• options {Object} an optional object containing settings used to adjust how the objects are merged
• returns {Object} the merged object

This merger has the following behavior:

• objects are iterated, and values are recursively merged
• a scalar (string, number, or boolean) in changes clobbers the value in base
• an array in changes clobbers the value in base

There currently are no defined options.

Customizing the Merge Algorithm

Once this libraries has identified which sections to use, it needs to merge the sections down into a single config. This final config is what is returned from read().

It does this by merging each section onto the root — later sections in the source are merged over earlier sections. Each section is merged using the merger function, which defaults to the one described above.

If you want to override how merging happens you can replace the one exported by this module. Your customer merger should have the same signature as the default merger.

var bigfig = require('bigfig');
var hoek = require('hoek');
var fig, config;
fig = new bifgig.Config(...);

bigfig.merger = function(base, changes, options) {
    // use hoek's implementation
    hoek.merge(base, changes, true, false);
    return base;
};

config = fig.read({...});

Optimizing a Custom Merger

While the merger should return the config to use, it doesn't need to be a newly created object. The base argument can be returned, modified or unmodified. (This can be a bit tricky — it's suggested to create a unit test which has a complex source, and calls read() multiple times on a single Config() object.)

Customizing the Match Algorithm

The context passed to read() is matched to each section in the source. This is done by calling the matcher, which defaults to the one described above.

The matcher is called with the context in the section, the context passed to read(), and should return a boolean indicating whether that section should be used.

If you want to override how matching happens you can replace the one exported by this module. Your custom matcher should have the same signature as the default merger.

var bigfig = require('bigfig');
var fig, config;
fig = new bifgig.Config(...);

bigfig.matcher = function(sectionContext, runContext, options) {
    // this toy matcher only matches sections based on the `env` key
    return sectionContext.env === runContext.env;
};

config = fig.read({...});

Optimizing Usage

If you want to trim the CPU, memory, and GC overhead of this library, here are some tricks:

• Create a Config object once (perhaps at app startup) and call read() multiple times. This library is specifically optimized for this usage pattern.

• The read() overhead depends on how many sections and how deeply nested the configs are. More deeply nested configs mean more time is spent in merging (which can also affect GC). Lots of sections means more time spent in matching, which is a simple algorithm on long-lived objects (little GC cost).

Deeply nested sections are optimized by the constructor and so don't affect performance. The following two examples have the same performance during read():

{
    memcache: {
        settings: {
            timeout: 1000,
            "__context?env=production": {
                timeout: 500
            }
        }
    }
}

{
    memcache: {
        settings: {
            timeout: 1000
        }
    },
    "__context?env=production": {
        memcache: {
            settings: {
                timeout: 500
            }
        }
    }
}

Ideas for Improvements

• better way to replace the matcher and merger
• customize special key prefix (instead of __context?)
• dimension values can have a hierarchy (for example, {env: 'prod/east'} matches both __context?env=prod and __context?env=prod/east)
• control the order in which the sections are merged (instead of order found in source)
• more sophisticated matcher API so that complex matchers can be highly optimized
• options inlined in the source (quivalent to options passed to the constructor)

Special Thanks

Bigfig is a direct decendent of ycb, which pioneered many of the ideas and priorities expressed in this library.

License

MIT License

Copyright 2015, Yahoo Inc.

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.