One simple way of parsing configs
conf.d directory~/.config supportconfight focuses on making application configuration easy to load, change, extend, generate and audit. It does so by allowing to use separated files for different topics, simplifying changes and new additions, without messing with already existing defaults or deleting or moving protected files.
This is achieved by using at least one config file (/etc/app/config.toml)
and an extra directory (/etc/app/conf.d) for extra files.
Those extra files are called droplets which consist in is a small config
file that is "dropped" into a conf.d directory where they will be parsed
and merged nicely into a single final configuration.
This approach is very common in Unix and used in several applications such as:
/etc/cron.d)/etc/profile.d)/etc/apt/sources.list.d)The idea is to "map reduce" configurations, by parsing all files in order, giving more relevance to the latest appearance and then merge them into a single config that holds all the data:
C₀ -- parse -----|
C₁ -- parse --|
C₂ -- parse --|-- merge --> C
⋮ |
Cₙ -- parse --|
The name of those files will determine the order in which they're parsed and the priority their values will have when merging. The last one wins.
Is specially good for externally managed configs or debian-packaged applications, avoiding clashes between installed files and generated configs, for changes that would stay forever unless manually merged.
>>> import confight
>>> confight.load_app('myapp')
{
"section": {
"key": "value"
}
}
The previous fragment got all the config files at /etc/myapp/config.toml and
within the /etc/myapp/conf.d directory and merged them into a single config.
# /etc/myapp/config.toml /etc/myapp/conf.d/00_first.json /etc/myapp/conf.d/99_second.ini
[section] { [section]
key = "base config" "section": { key = value
"key": "not this"
}
}
Default file locations for an application named myapp would be at:
/etc/myapp/config.toml/etc/myapp/conf.d/*User custom configurations would be read (if any) from:
~/.config/myapp/config.toml~/.config/myapp/conf.d/*See the examples section for more information on how to use these functions.
The load family of functions take a list of names, files or directories to
easily parse and merge a related set of configurations:
confight.load_app('myapp')
confight.load_user_app('myapp')
confight.load_paths(['/path/to/config', '/path/to/dir'])
confight.load(['/path/to/config.toml', '/path/to/dir/droplet.toml'])
Each function offers different parameters to improve the ease of use.
The extension of the configuration file can be given with the extension
parameter. For instance, load_app('myapp', extension='json') would look for
the /etc/myapp/config.json file.
All files in the conf.d directory are read by default regardless the
extension. To enforce that only .extension files are read, add the
force_extension flag.
Some formats are builtin in the default installation and some others are optional and must be declared when installing confight.
The list of builtin file formats:
The list of optional file formats:
In order to install confight with optional formats see installation with optional features.
Given a path to an existing configuration file, it will be loaded in memory
using basic types (string, int, float, list, dict).
The given file can be in one of the allowed formats. For a complete list see
the confight.FORMATS list.
confight.parse('/path/to/config', format='toml')
When no format is given, it tries to guess by looking at file extensions:
confight.parse('/path/to/config.json') # will gess json format
You can see the list of all available extensions at confight.FORMAT_EXTENSIONS.
A custom parsing can be provided by passing a parser function to the load
family of functions, matching the signature:
def parser(path, format=None)
The function takes a filesystem path and a format and the result should
be a single dictionary with all the loaded data. When format is None the
parser is expected to guess it.
Given a list of parsed configs in order, merge them into a single one. For values that appears several times, the last one wins.
Sections and subsections are recursively merged, keeping all keys along the way and overriding the ones in more than one file with the latest appearance.
A custom merging can be provided by passing a merger function to the load
family of functions, matching the signature:
def merger(configs)
The function takes a list of dictionaries containing the parsed configuration in ascending order of priority. It should return a single dictionary with all the configuration.
The default behaviour is that all files at the conf.d directory will be
opened, in lexicographical order, and parsed.
A custom config locator can be provided by passing a finder function to the
load family of functions, matching the signature:
def finder(path)
The function takes a filesystem path (a conf.d directory supposedly) and
returns a list of paths to config files in the desired order of parsing and
merging, this is from less to more priority for their values.
Load application config from the default locations by using the load_app
function which will look by default at the /etc/myapp/config.toml and
configuration directory at /etc/myapp/conf.d:
# /etc/myapp/config.toml # /etc/myapp/conf.d/production.toml
user = myapp password = aX80@klj
password = guest
>>> confight.load_app('myapp')
{
"user": "myapp",
"password": "aX80@klj"
}
Allow the user to override the default value when wanting to use a different configuration. When None is given, the default is used:
import argparse
import confight
parser = argparse.ArgumentParser()
parser.add_argument('--config', default=None)
parser.add_argument('--config-dir', default=None)
args = parser.parse_args()
config = confight.load_app('myapp',
file_path=args.config,
dir_path=args.config_dir)
If the application supports user configuration the function load_user_app
might come handy as it will first load the regular app config and then the one
defined in the user directory ~/.config/myapp/config.toml and
~/.config/myapp/conf.d/*:
# /etc/myapp/config.toml # ~/.config/myapp/conf.d/mysettings.toml
url = http://teg.avature.net password = Avature123!
>>> confight.load_user_app('myapp')
{
"url": "http://teg.avature.net",
"password": "Avature123!"
}
To ignore config file extensions, set a format and all files will be parsed using such:
# /etc/myapp/config.toml # /etc/myapp/config.d/extra
name = test name = erebus
>>> confight.load_app('myapp', format='toml')
{
"name": "erebus"
}
To load configs from a dev or debug location use the prefix option.
This will change the base to calculate default paths.
# Loads from ./configs/config.toml and ./configs/config.d/*
>>> confight.load_app('myapp', prefix='./configs')
The user_prefix option can be used altogether for user config files:
# Loads from regular places and ./user/config.toml and ./user/config.d/*
>>> confight.load_user_app('myapp', user_prefix='./user')
Added in version 1.0
confight allows to inspect configuration from the command line.
By using the confight command it would load the myapp configuration from it's default places and display the output in toml format:
confight show myapp
This allows to preview the resulting config for an application after all merges have been resolved. It can come handy when figuring out what the application has loaded or to debug complex config scenarios.
By passing the --verbose INFO interesting data such as all visited files is
listed.
Added in version 0.3
usage: confight [-h] [--version] [-v {DEBUG,INFO,WARNING,ERROR,CRITICAL}]
{show} ...
One simple way of parsing configs
positional arguments:
{show}
optional arguments:
-h, --help show this help message and exit
--version show program's version number and exit
-v {DEBUG,INFO,WARNING,ERROR,CRITICAL}, --verbose {DEBUG,INFO,WARNING,ERROR,CRITICAL}
Logging level default: ERROR ['DEBUG', 'INFO',
'WARNING', 'ERROR', 'CRITICAL']
Install it via pip using:
pip install confight
Also with yaml support using optional features:
pip install confight[yaml]
Similarly, for hcl support:
pip install confight[hcl]
Run application tests
tox
Install the application and run tests in development:
pip install -e .
python -m pytest
2.0 (2023-12-13)
1.4.0 (2023-12-12)
[ Federico Fapitalle ]
[ Frank Lenormand ]
ruamel.yaml APIpyhcl package version1.3.2 (2023-10-25)
ruamel.yaml1.3.1 (2020-04-28)
1.3 (2019-09-19)
1.2.3 (2019-02-25)
1.2.2 (2019-02-19)
1.2.1 (2019-02-19)
1.2 (2019-02-14)
1.1.1 (2019-01-31)
[ javier.lasheras ]
1.1 (2019-01-29)
1.0 (2018-06-26)
0.3 (2018-06-14)
0.2.2 (2018-04-13)
0.2.1 (2018-04-09)
0.2 (2018-04-04)
0.1.1 (2018-04-03)
0.1.0 (2018-03-27)
0.0.1 (2018-03-27)
FAQs
Common config loading for Python and the command line
We found that confight demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 2 open source maintainers collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
PyPI confirms no security flaws were exploited in the Ultralytics supply chain attack and highlights improvements for safer package publishing.
Security News
Research
The Socket Research Team breaks down a malicious wrapper package that uses obfuscation to harvest credentials and exfiltrate sensitive data.
Research
Security News
Attackers used a malicious npm package typosquatting a popular ESLint plugin to steal sensitive data, execute commands, and exploit developer systems.