parse_it
A python library for parsing multiple types of config files, envvars and command line arguments that takes the headache out of setting app configurations.
Github actions CI unit tests & auto PyPi push status:
Code coverage:
Install
First install parse_it, for Python 3.6 & higher this is simply done using pip:
pip install parse_it
If your using a Python 3.4 or older you will require the typing
backported package as well, this is done with the following optional install:
pip install parse_it[typing]
How to use
from parse_it import ParseIt
parser = ParseIt()
my_config_key = parser.read_configuration_variable("my_config_key")
By default all configuration files will be assumed to be in the workdir but if you want you can also easily set it to look in all subfolders recursively:
from parse_it import ParseIt
parser = ParseIt(config_location="/etc/my_config_folder", recurse=True)
my_config_key = parser.read_configuration_variable("my_int")
By default parse_it will look for the configuration options in the following order & will return the first one found:
cli_args
- command line arguments that are passed in the following format --key value
env_vars
- environment variables, you can also use envvars
as an alias for itenv
- .env formatted files, any file ending with a .env extension in the configuration folder is assumed to be thisjson
- JSON formatted files, any file ending with a .json extension in the configuration folder is assumed to be thisyaml
- YAML formatted files, any file ending with a .yaml extension in the configuration folder is assumed to be thisyml
- YAML formatted files, any file ending with a .yml extension in the configuration folder is assumed to be thistoml
- TOML formatted files, any file ending with a .toml extension in the configuration folder is assumed to be thistml
- TOML formatted files, any file ending with a .tml extension in the configuration folder is assumed to be thishcl
- HCL formatted files, any file ending with a .hcl extension in the configuration folder is assumed to be thistf
- HCL formatted files, any file ending with a .tf extension in the configuration folder is assumed to be thisconf
- INI formatted files, any file ending with a .conf extension in the configuration folder is assumed to be thiscfg
- INI formatted files, any file ending with a .cfg extension in the configuration folder is assumed to be thisini
- INI formatted files, any file ending with a .ini extension in the configuration folder is assumed to be thisxml
- XML formatted files, any file ending with a .xml extension in the configuration folder is assumed to be this- configuration default value - every configuration value can also optionally be set with a default value
- global default value - the parser object also has a global default value which can be set
if multiple files of the same type exists in the same folder parse_it will look in all of them in alphabetical order before going to the next type,
You can decide on using your own custom order of any subset of the above options (default values excluded, they will always be last):
from parse_it import ParseIt
parser = ParseIt(config_type_priority=["env_vars", "yaml", "yml", "json"])
The global default value by default is None but if needed it's simple to set it:
from parse_it import ParseIt
parser = ParseIt()
my_config_key = parser.read_configuration_variable("my_undeclared_key")
parser = ParseIt(global_default_value="my_default_value")
my_config_key = parser.read_configuration_variable("my_undeclared_key")
parse_it will by default attempt to figure out the type of value returned so even in the case of envvars, cli args & INI files you will get strings/dicts/etc:
from parse_it import ParseIt
import os
os.environ["MY_INT"] = "123"
os.environ["MY_LIST"] = "['first_item', 'second_item', 'third_item']"
os.environ["MY_DICT"] = "{'key': 'value'}"
parser = ParseIt()
my_config_key = parser.read_configuration_variable("MY_INT")
my_config_key = parser.read_configuration_variable("MY_LIST")
my_config_key = parser.read_configuration_variable("MY_DICT")
parser = ParseIt(type_estimate=False)
my_config_key = parser.read_configuration_variable("MY_INT")
my_config_key = parser.read_configuration_variable("MY_LIST")
my_config_key = parser.read_configuration_variable("MY_DICT")
As envvars recommended syntax is to have all keys be UPPERCASE which is diffrent then all the rest of the configuration files parse_it will automatically change any needed config value to be in ALL CAPS when looking at envvars for the matching value but if needed you can of course disable that feature:
from parse_it import ParseIt
import os
os.environ["MY_STRING"] = "UPPER"
os.environ["my_string"] = "lower"
parser = ParseIt()
my_config_key = parser.read_configuration_variable("my_string")
parser = ParseIt(force_envvars_uppercase=False)
my_config_key = parser.read_configuration_variable("my_string")
You can also easily add a prefix to all envvars (note that force_envvars_uppercase
will also affect the given prefix):
from parse_it import ParseIt
import os
os.environ["PREFIX_MY_INT"] = "123"
parser = ParseIt(envvar_prefix="prefix_")
my_config_key = parser.read_configuration_variable("my_int")
You can also set a default value on a per configuration key basis:
from parse_it import ParseIt
parser = ParseIt()
my_config_key = parser.read_configuration_variable("my_undeclared_key", default_value="my_value")
You can also declare a key to be required (disabled by default) so it will raise a ValueError if not declared by the user anywhere:
from parse_it import ParseIt
parser = ParseIt()
my_config_key = parser.read_configuration_variable("my_undeclared_key", required=True)
While generally not a good idea sometimes you can't avoid it and will need to use a custom non standard file suffix, you can add a custom mapping of suffixes to any of the supported file formats as follows (note that config_type_priority
should also be set to configure the priority of said custom suffix):
from parse_it import ParseIt
parser = ParseIt(config_type_priority=["env_vars", "custom_yaml_suffix", "yaml", "yml", "json"], custom_suffix_mapping={"yaml": ["custom_yaml_suffix"]})
You might sometimes want to check that the enduser passed to your config a specific type of variable, parse_it allows you to easily check if a value belongs to a given list of types by setting allowed_types
which will then raise a TypeError if the value type given is not in the list of allowed_types
, by default this is set to None so no type ensuring takes place:
from parse_it import ParseIt
import os
os.environ["ONLY_INTGERS_PLEASE"] = "123"
parser = ParseIt()
my_config_key = parser.read_configuration_variable("only_intgers_please")
my_config_key = parser.read_configuration_variable("only_intgers_please", allowed_types=[int])
my_config_key = parser.read_configuration_variable("only_intgers_please", allowed_types=[str, dict, list, None])
Sometimes you'll need a lot of configuration keys to have the same parse_it configuration params, rather then looping over them yourself this can be achieved with the read_multiple_configuration_variables
function that you will give it a list of the configuration keys you want & will apply the same configuration to all and return you a dict with the key/value of the configurations back.
from parse_it import ParseIt
parser = ParseIt()
my_config_key = parser.read_multiple_configuration_variables(["my_first_config_key", "my_second_config_key"], default_value="default_value", required=False, allowed_types=[str, list, dict, int])
You can also read a single file rather then a config directory.
from parse_it import ParseIt
parser = ParseIt(config_location="/etc/my_config_folder/my_config.json")
my_config_key = parser.read_configuration_variable("my_int")
Another option is to read all configurations from all valid sources into a single dict that will include the combined results of all of them (by combined it means it will return only the highest priority of each found key & will combine different keys from different sources into a single dict), this provides less flexibility then reading the configuration variables one by one and is a tad (but just a tad) slower but for some use cases is simpler to use:
from parse_it import ParseIt
parser = ParseIt()
my_config_dict = parser.read_all_configuration_variables()
my_config_dict = parser.read_all_configuration_variables(default_value={"my_key": "my_default_value", "my_other_key": "my_default_value"}, required=["my_required_key","my_other_required_key"], allowed_types={"my_key": [str, list, dict, int], "my_other_key": [str, list, dict, int]})
It has also become a common practice to divide envvar keys by a divider character (usually _
) and nest then as subdicts, this assists in declaring complex dictionaries subkeys with each of them being given it's own key, parse_it supports this option as well by setting the envvar_divider
variable when declaring the parse_it object (disabled by default):
from parse_it import ParseIt
import os
os.environ["NEST1_NEST2_NEST3"] = "123"
parser = ParseIt(envvar_divider="_")
my_config_dict = parser.read_all_configuration_variables()
You can define which values should be considered None type. Default is {"", "null", "none"}
from parse_it import ParseIt
import os
os.environ["my_config_key1"] = "Nothing"
os.environ["my_config_key2"] = "null"
parser = ParseIt(none_values={"Nothing", "null"})
my_config_key1 = parser.read_configuration_variable("my_config_key1")
my_config_key2 = parser.read_configuration_variable("my_config_key2")