onion_config
![PyPI - Python Version](https://img.shields.io/pypi/pyversions/onion-config?logo=Python)
onion_config
is a python package that allows for easy configuration management. It allows for loading and validating configuration data from environment variables and config files in JSON and YAML formats.
Pydantic
based custom config package for python projects.
Features
- Main config based on Pydantic schema - https://pypi.org/project/pydantic
- Load environment variables - https://pypi.org/project/python-dotenv
- Load from multiple configs directories
- Load configs from YAML and JSON files
- Update the default config with additional configurations (
extra_dir
directory) - Pre-load hook function to modify config data before loading and validation
- Validate config values with Pydantic validators
- Config as dictionary or Pydantic model (with type hints)
- Pre-defined base config schema for common config (
BaseConfig
) - Base for custom config loader (
ConfigLoader
)
Installation
1. Prerequisites
- Python (>= v3.8)
- PyPi (>= v23)
2. Install onion-config package
Choose one of the following methods to install the package [A ~ F]:
A. [RECOMMENDED] Install from PyPi
pip install -U onion-config[pydantic-v1]
pip install -U onion-config[pydantic-settings]
B. Install latest version from GitHub
pip install git+https://github.com/bybatkhuu/module.python-config.git[pydantic-v1]
pip install git+https://github.com/bybatkhuu/module.python-config.git[pydantic-settings]
C. Install from pre-built release files
- Download
.whl
or .tar.gz
file from releases - https://github.com/bybatkhuu/module.python-config/releases - Install with pip:
pip install ./onion_config-[VERSION]-py3-none-any.whl[pydantic-v1]
pip install ./onion_config-[VERSION].tar.gz[pydantic-v1]
pip install ./onion_config-[VERSION]-py3-none-any.whl[pydantic-settings]
pip install ./onion_config-[VERSION].tar.gz[pydantic-settings]
D. Install from source code by building package
git clone https://github.com/bybatkhuu/module.python-config.git onion_config
cd ./onion_config
pip install -U pip build
python -m build
_VERSION=$(./scripts/get-version.sh)
pip install ./dist/onion_config-${_VERSION}-py3-none-any.whl[pydantic-v1]
pip install ./dist/onion_config-${_VERSION}.tar.gz[pydantic-v1]
pip install ./dist/onion_config-${_VERSION}-py3-none-any.whl[pydantic-settings]
pip install ./dist/onion_config-${_VERSION}.tar.gz[pydantic-settings]
E. Install with pip editable development mode (from source code)
git clone https://github.com/bybatkhuu/module.python-config.git onion_config
cd ./onion_config
pip install -e .[pydantic-v1]
pip install -e .[pydantic-settings]
F. Manually add to PYTHONPATH (not recommended)
git clone https://github.com/bybatkhuu/module.python-config.git onion_config
cd ./onion_config
pip install -r ./requirements.txt
pip install -r ./requirements.pydantic-v2.txt
export PYTHONPATH="${PWD}:${PYTHONPATH}"
Usage/Examples
To use onion_config
, import the ConfigLoader
class from the package:
from onion_config import ConfigLoader, BaseConfig
You can create an instance of ConfigLoader
with auto_load
flag. This will automatically load configuration data from environment variables and config files located in the default directory ('./configs'
). The configuration data can then be accessed via the config
property of the ConfigLoader
instance:
config: BaseConfig = ConfigLoader(auto_load=True).config
Simple
.env
ENV=production
configs/1.base.yml
:
env: test
app:
name: "My App"
version: "0.0.1"
nested:
key: "value"
configs/2.extra.yml
:
app:
name: "New App"
nested:
some: "value"
description: "Description of my app."
another_val:
extra: 1
main.py
import pprint
from loguru import logger
try:
import pydantic_settings
_has_pydantic_settings = True
except ImportError:
_has_pydantic_settings = False
from onion_config import ConfigLoader, BaseConfig
class ConfigSchema(BaseConfig):
env: str = "local"
try:
config: ConfigSchema = ConfigLoader(config_schema=ConfigSchema).load()
except Exception:
logger.exception("Failed to load config:")
exit(2)
if __name__ == "__main__":
logger.info(f"All: {config}")
logger.info(f"App name: {config.app['name']}")
if _has_pydantic_settings:
logger.info(f"Config:\n{pprint.pformat(config.model_dump())}\n")
else:
logger.info(f"Config:\n{pprint.pformat(config.dict())}\n")
Run the examples/simple
:
cd ./examples/simple
python ./main.py
Output:
2023-09-01 00:00:00.000 | INFO | __main__:<module>:29 - All: env='production' another_val={'extra': 1} app={'name': 'New App', 'version': '0.0.1', 'nested': {'key': 'value', 'some': 'value'}, 'description': 'Description of my app.'}
2023-09-01 00:00:00.000 | INFO | __main__:<module>:30 - App name: New App
2023-09-01 00:00:00.000 | INFO | __main__:<module>:35 - Config:
{'another_val': {'extra': 1},
'app': {'description': 'Description of my app.',
'name': 'New App',
'nested': {'key': 'value', 'some': 'value'},
'version': '0.0.1'},
'env': 'production'}
Advanced
.env.base
:
ENV=development
DEBUG=true
APP_NAME="Old App"
ONION_CONFIG_EXTRA_DIR="extra_configs"
.env.prod
:
ENV=production
APP_NAME="New App"
APP_SECRET="my_secret"
configs/config.yml
:
env: local
app:
name: "My App"
port: 9000
bind_host: "0.0.0.0"
version: "0.0.1"
ignore_val: "Ignore me"
logger:
output: "file"
configs/logger.json
:
{
"logger": {
"level": "info",
"output": "stdout"
}
}
configs_2/config.yml
:
extra:
config:
key1: 1
configs_2/config_2.yml
:
extra:
config:
key2: 2
extra_configs/extra.json
:
{
"extra": {
"type": "json"
}
}
schema.py
:
from enum import Enum
from typing import Union
import pydantic
from pydantic import Field, SecretStr
_has_pydantic_settings = False
if "2.0.0" <= pydantic.__version__:
try:
from pydantic_settings import SettingsConfigDict
_has_pydantic_settings = True
except ImportError:
pass
from onion_config import BaseConfig
class EnvEnum(str, Enum):
LOCAL = "local"
DEVELOPMENT = "development"
TEST = "test"
DEMO = "demo"
STAGING = "staging"
PRODUCTION = "production"
class AppConfig(BaseConfig):
name: str = Field("App", min_length=2, max_length=32)
bind_host: str = Field("localhost", min_length=2, max_length=128)
port: int = Field(8000, ge=80, lt=65536)
secret: SecretStr = Field(..., min_length=8, max_length=64)
version: str = Field(..., min_length=5, max_length=16)
description: Union[str, None] = Field(None, min_length=4, max_length=64)
if _has_pydantic_settings:
model_config = SettingsConfigDict(extra="ignore", env_prefix="APP_")
else:
class Config:
extra = "ignore"
env_prefix = "APP_"
class ConfigSchema(BaseConfig):
env: EnvEnum = Field(EnvEnum.LOCAL)
debug: bool = Field(False)
app: AppConfig = Field(...)
config.py
:
from loguru import logger
from onion_config import ConfigLoader
from schema import ConfigSchema
def _pre_load_hook(config_data: dict) -> dict:
config_data["app"]["port"] = "80"
config_data["extra_val"] = "Something extra!"
return config_data
config = None
try:
_config_loader = ConfigLoader(
config_schema=ConfigSchema,
configs_dirs=["configs", "configs_2", "/not_exists/path/configs_3"],
env_file_paths=[".env", ".env.base", ".env.prod"],
pre_load_hook=_pre_load_hook,
config_data={"base": "start_value"},
warn_mode="ALWAYS",
)
config: ConfigSchema = _config_loader.load()
except Exception:
logger.exception("Failed to load config:")
exit(2)
main.py
:
import pprint
from loguru import logger
try:
import pydantic_settings
_has_pydantic_settings = True
except ImportError:
_has_pydantic_settings = False
from config import config
if __name__ == "__main__":
logger.info(f"All: {config}")
logger.info(f"ENV: {config.env}")
logger.info(f"DEBUG: {config.debug}")
logger.info(f"Extra: {config.extra_val}")
logger.info(f"Logger: {config.logger}")
logger.info(f"App: {config.app}")
logger.info(f"Secret: '{config.app.secret.get_secret_value()}'\n")
if _has_pydantic_settings:
logger.info(f"Config:\n{pprint.pformat(config.model_dump())}\n")
else:
logger.info(f"Config:\n{pprint.pformat(config.dict())}\n")
try:
config.app.port = 8443
except Exception as e:
logger.error(f"{e}\n")
Run the examples/advanced
:
cd ./examples/advanced
python ./main.py
Output:
2023-09-01 00:00:00.000 | INFO | onion_config._base:load:143 - Loading all configs...
2023-09-01 00:00:00.000 | WARNING | onion_config._base:_load_dotenv_file:201 - '/home/user/workspaces/projects/onion_config/examples/advanced/.env' file is not exist!
2023-09-01 00:00:00.000 | WARNING | onion_config._base:_load_configs_dir:257 - '/not_exists/path/configs_3' directory is not exist!
2023-09-01 00:00:00.000 | SUCCESS | onion_config._base:load:171 - Successfully loaded all configs!
2023-09-01 00:00:00.000 | INFO | __main__:<module>:19 - All: env=<EnvEnum.PRODUCTION: 'production'> debug=True app=AppConfig(name='New App', bind_host='0.0.0.0', port=80, secret=SecretStr('**********'), version='0.0.1', description=None) extra={'config': {'key1': 1, 'key2': 2}, 'type': 'json'} extra_val='Something extra!' logger={'output': 'stdout', 'level': 'info'} base='start_value'
2023-09-01 00:00:00.000 | INFO | __main__:<module>:20 - ENV: production
2023-09-01 00:00:00.000 | INFO | __main__:<module>:21 - DEBUG: True
2023-09-01 00:00:00.000 | INFO | __main__:<module>:22 - Extra: Something extra!
2023-09-01 00:00:00.000 | INFO | __main__:<module>:23 - Logger: {'output': 'stdout', 'level': 'info'}
2023-09-01 00:00:00.000 | INFO | __main__:<module>:24 - App: name='New App' bind_host='0.0.0.0' port=80 secret=SecretStr('**********') version='0.0.1' description=None
2023-09-01 00:00:00.000 | INFO | __main__:<module>:25 - Secret: 'my_secret'
2023-09-01 00:00:00.000 | INFO | __main__:<module>:30 - Config:
{'app': {'bind_host': '0.0.0.0',
'description': None,
'name': 'New App',
'port': 80,
'secret': SecretStr('**********'),
'version': '0.0.1'},
'base': 'start_value',
'debug': True,
'env': <EnvEnum.PRODUCTION: 'production'>,
'extra': {'config': {'key1': 1, 'key2': 2}, 'type': 'json'},
'extra_val': 'Something extra!',
'logger': {'level': 'info', 'output': 'stdout'}}
2023-09-01 00:00:00.000 | ERROR | __main__:<module>:36 - "AppConfig" is immutable and does not support item assignment
Running Tests
To run tests, run the following command:
pip install -r ./requirements.txt
pip install -r ./requirements.pydantic-v2.txt
pip install -r ./requirements.test.txt
python -m pytest -v
FAQ
What is the order of loading config?
Load order:
- Load all dotenv files from
env_file_paths
into environment variables. - Check if required environment variables exist or not.
- Load all config files from
configs_dirs
into config_data
. - Load extra config files from
extra_dir
into config_data
. - Execute
pre_load_hook
method to modify config_data
. - Init
config_schema
with config_data
into final config
.
Environment Variables
You can use the following environment variables inside .env.example
file:
ONION_CONFIG_EXTRA_DIR="./extra_configs"
Documentation
References