lazyConfig

lazyConfig

lazily loading and overriding configuration

lazyConfig is an opinionated configuration provider. It can load .yml, .yaml, .json and .toml configuration files out of the box but can be extended with custom loaders. Opinionated since it requires you to structure your configuration in a certain way in order to work.

Why lazyConfig?

see Motivation

Usage

In order to load your configuration folder into a dict like structure, simply point lazyConfig to the folder

import lazyConfig

config = lazyConfig.from_path(
    config='/path/to/default/config',
    override=[
        '/path/to/override',
        '/path/to/another/override/',
        ...
    ]
)

# or with environment variables:

os.environ['CONFIG'] = '/path/to/default/config'
os.environ['CONFIG_OVERRIDE'] = f'/path/to/override/{os.pathsep}/path/to/another/override/'
config = lazyConfig.from_env()

You can change the envrionment variable names by passing their names, e.g.

lazyConfig.from_env(config='DEFAULT_CONFIG', override='CONFIG')

Files are overriden left to right (i.e. the last argument has priority)

you could mix and match using Config.from_path and os.environ['ENV_VAR']

Add override

If you are using an command line interface parser (e.g. argparse) to generate a dictionary for overriding configuration, you might want to add an additional override. You can use

config.add_override(argparse_results)

where argparse_results is of type Mapping

use vars() to obtain a dict from an arparse.Namespace which is the return type of an arparse parser

arparse uses None to indicate missing configuration. So add_override ignores None by default. If you provide the parameter none_can_override=True, you can remove configuration with None values.

Override Restrictions

You can only override keys which exist in the default configuration. This requirement ensures that the default configuration documents all possible configuration options. Use the value None in the default configuration for settings you do not want to select a default for. You can obtain a dict without these entries with

config.as_dict(strip_none=True)

strip_none=True is the default, set it to false if you want a dict including all None values

Assumptions about the file structure

Filenames are used as keys in the config dict, so without any caveats you would be forced to have a minimum depth of 1 for any configuration.

This might not make sense for some high level, flat configuration. For this reason there is a special filename __config__ (.yml,.yaml, .json,...), which allows you to define top level keys and values.

Example

config
    database
        configuration.yml
        __config__.yml
    app.yml
    __config__.yml

config/__config__.yml:

name: my-app
author: ME!
version: -1.0

app.yml:

primary_color: 'blue'
secondary_color: 'green'

config/database/__config__.yml:

connection:
    hosts:
        - {host: "myElasticsearchServer", port: 9200}
    timeout: 6000

config/database/configuration.yml:

indices:
    index1: {...}
    index2: {...}
pipelines:
    pipeline1: {...}

would be loaded as the following dictionary (formatted in yml for readability):

name: my-app
author: ME!
version: -1.0
app:
    primary_color: 'blue'
    secondary_color: 'green'
database:
    connection:
        hosts:
            - {host: "myElasticsearchServer", port: 9200}
        timeout: 6000
    configuration:
        indices:
            index1: {...}
            index2: {...}
        pipelines:
            pipeline1: {...}

Attribute access

You could then access index1 with:

config.database.configuration.indices.index1

Python does not allow attributes to start with a number. So

config.1attribute2breakthings

will not work. In these cases you will have to use

config['1attribute2breakthings']

Duplicate Keys

Duplicate Keys are not allowed. lazyConfig tries to find keys in

1. Keyfiles (__config__)
2. Filenames
3. Directories

in that order! It will not keep looking whether or not there is a duplicate and thus ignore a directory if a file with the same name (sans extension) or a key with the same name in the keyfile exists.

There might be a configuration validation function in the future to check for duplicate keys (in debug mode or on manual call)

Lists

Lists are overriden not extended. If the default configuration has the same key for a list as an override, then the default list is ignored and the override is used.

Extending Lists would mean that you can not remove elements with a configuration override. As this is not desirable you will just have to duplicate lists in its entirety if you want to change them (even if only slightly)

Since numbers are discouraged anyway, you can define Directory Lists like this:

config
    list
        0.yml
        1.yml
        2.yml
        3.yml

which is then converted to

config:
    list:
        - {content from 0.yml}
        - {content from 1.yml}
        - {content from 2.yml}

Directory Lists must start with 0 and end with n-1 where n is the number of files in the directory.

It is currently not possible to create a list of directories (instead of files). This might become a feature in a future version if requested

Security

Using pyYAML.unsafe_load(), lazyConfig is currently not meant for external data. Pass a custom_extension_loader to the factory method:

lazyConfig.from_path('path/to/config', custom_extension_loader={
    '.yml': yaml.safe_load,
    '.yaml': yaml.safe_load
})

This overrides the yaml loader while leaving the other loaders in place. To remove a default loader, override the extension with a Falsy value (e.g. None)

Future Features

things which I might get around to do some time:

• Configuration Validation
• Horizontal override (Templates, e.g. for indices)
• Configuration Setter

  Since it is not obvious where to set directory/file boundaries inside the dictionary, this is not as trivial as it might seem at first. It might be necessary to provide the grouping level as an argument.

• generate JSON Schema from default configuration

Versions

• 0.5 proper none handling
• 0.4 add_overwrite
• 0.3 as_primitive, as_dict, as_list, laziness modes and custom loader, TOML support
• 0.2.2 fix equality
• 0.2.1 fix broken iterator
• 0.2 Config implements Mapping
• 0.1.1 handle NULL environment variables
• 0.1: basic proof of concept