pydantic-file-secrets

pydantic-file-secrets 🔑

Use file secrets in nested pydantic-settings models instead of built-in SecretsSettingsSource

This project is inspired by discussions in Pydantic Settings repository and proposes solution to #30 and #154.

Features

• Plain or nested directory layout: secrets/dir__key or secrets/dir/key
• Respects env_prefix, env_nested_delimiter and other config options
• Implements config options secrets_prefix, secrets_nested_delimiter and more to configure secrets and env vars independently
• Drop-in replacement of standard SecretsSettingsSource
• Pure Python thin wrapper over standard EnvSettingsSource
• No third party dependencies except pydantic-settings
• Fully typed
• 100% test coverage

Installation

$ pip install pydantic-file-secrets

Motivation

Nested Pydantic config can contain nested models with secret entries, as well as secrets in top level config. In dockerized environment, these entries may be read from file system, e.g. /run/secrets when using Docker Secrets:

from pydantic import BaseModel, Secret
from pydantic_settings import BaseSettings, SettingsConfigDict

class DbSettings(BaseModel):
    user: str
    passwd: Secret[str]  # secret in nested model

class Settings(BaseSettings):
    app_key: Secret[str]  # secret in root model
    db: DbSettings

    model_config = SettingsConfigDict(
        secrets_dir='/run/secrets',
    )

Usage

Plain secrets directory layout

📂 secrets
├── 📄 app_key
└── 📄 db__passwd

from pydantic import BaseModel, SecretStr
from pydantic_file_secrets import FileSecretsSettingsSource, SettingsConfigDict
from pydantic_settings import BaseSettings
from pydantic_settings.sources import PydanticBaseSettingsSource


class DbSettings(BaseModel):
    passwd: SecretStr


class Settings(BaseSettings):
    app_key: SecretStr
    db: DbSettings

    model_config = SettingsConfigDict(
        secrets_dir='secrets',
        secrets_nested_delimiter='__',
    )

    @classmethod
    def settings_customise_sources(
        cls,
        settings_cls: type[BaseSettings],
        init_settings: PydanticBaseSettingsSource,
        env_settings: PydanticBaseSettingsSource,
        dotenv_settings: PydanticBaseSettingsSource,
        file_secret_settings: PydanticBaseSettingsSource,
    ) -> tuple[PydanticBaseSettingsSource, ...]:
        return (
            init_settings,
            env_settings,
            dotenv_settings,
            FileSecretsSettingsSource(file_secret_settings),
        )

>>> Settings().model_dump()
{'app_key': SecretStr('**********'), 'db': {'passwd': SecretStr('**********')}}

Nested secrets directory layout

Config option secrets_nested_delimiter overrides env_nested_delimiter for files. In particular, this allows to use nested directory layout along with environmemt variables for other non-secret settings:

📂 secrets
├── 📄 app_key
└── 📂 db
    └── 📄 passwd

    model_config = SettingsConfigDict(
        secrets_dir='secrets',
        secrets_nested_subdir=True,
    )

>>> Settings().model_dump()
{'app_key': SecretStr('**********'), 'db': {'passwd': SecretStr('**********')}}

Multiple secrets_dir

📂 secrets
├── 📂 layer1
│   └── 📄 app_key
└── 📂 layer2
    └── 📄 db__passwd

    model_config = SettingsConfigDict(
        secrets_dir=['secrets/layer1', 'secrets/layer2'],
        secrets_nested_delimiter='__',
    )

>>> Settings().model_dump()
{'app_key': SecretStr('**********'), 'db': {'passwd': SecretStr('**********')}}

Experimental syntactic sugar 🧪

[!CAUTION] This syntax may change at any time. Pin current pydantic-file-secrets version if decided to use it.

Few important things to note:

• @with_builtin_sources decorator enables NamedTuple argument src: BuiltinSources encapsulating default builtins settings sources
• BaseSource alias is shorter than PydanticBaseSettingsSource and is easier to use in type hints
• settings_cls was removed from settings_customise_sources signature: cls seems to be sufficient

from pydantic import BaseModel, SecretStr
from pydantic_file_secrets import (
    BaseSource,
    BuiltinSources,
    FileSecretsSettingsSource,
    SettingsConfigDict,
    with_builtin_sources,
)
from pydantic_settings import BaseSettings


class DbSettings(BaseModel):
    passwd: SecretStr


class Settings(BaseSettings):
    app_key: SecretStr
    db: DbSettings

    model_config = SettingsConfigDict(
        secrets_dir='secrets',
        secrets_nested_delimiter='__',
    )

    @classmethod
    @with_builtin_sources
    def settings_customise_sources(cls, src: BuiltinSources) -> tuple[BaseSource, ...]:
        return (
            src.init_settings,
            src.env_settings,
            src.dotenv_settings,
            FileSecretsSettingsSource(src.file_secret_settings),
        )

>>> Settings().model_dump()
{'app_key': SecretStr('**********'), 'db': {'passwd': SecretStr('**********')}}

Configuration options

secrets_dir

Path to secrets directory. Same as SecretsSettingsSource.secrets_dir if str or Path. If list, the last match wins. If secrets_dir is passed in both source constructor and model config, values are not merged (constructor takes priority).

secrets_dir_missing

If secrets_dir does not exist, original SecretsSettingsSource issues a warning. However, this may be undesirable, for example if we don't mount Docker Secrets in e.g. dev environment. Now you have a choice:

• 'ok' — do nothing if secrets_dir does not exist
• 'warn' (default) — print warning, same as SecretsSettingsSource
• 'error' — raise SettingsError

If multiple secrets_dir passed, the same secrets_dir_missing action applies to each of them.

secrets_dir_max_size

Limit the size of secrets_dir for security reasons, defaults to SECRETS_DIR_MAX_SIZE equal to 16 MiB.

FileSecretsSettingsSource is a thin wrapper around EnvSettingsSource, which loads all potential secrets on initialization. This could lead to MemoryError if we mount a large file under secrets_dir.

If multiple secrets_dir passed, the limit applies to each directory independently.

secrets_case_sensitive

Same as case_sensitive, but works for secrets only. If not specified, defaults to case_sensitive.

secrets_nested_delimiter

Same as env_nested_delimiter, but works for secrets only. If not specified, defaults to env_nested_delimiter. This option is used to implement nested secrets directory layout and allows to do even nastier things like /run/secrets/model/delim/nested1/delim/nested2.

secrets_nested_subdir

Boolean flag to turn on nested secrets directory mode, False by default. If True, sets secrets_nested_delimiter to os.sep. Raises SettingsError if secrets_nested_delimiter is already specified.

secrets_prefix

Secret path prefix, similar to env_prefix, but works for secrets only. Defaults to env_prefix if not specified. Works in both plain and nested directory modes, like '/run/secrets/prefix_model__nested' and '/run/secrets/prefix_model/nested'.

Not supported config options

Some config options that are declared in SecretsSettingsSource interface are actually not working and are not supported in FileSecretsSettingsSource:

• env_ignore_empty
• env_parse_none_str
• env_parse_enums

However, we make sure that the behaviour of FileSecretsSettingsSource matches SecretsSettingsSource to provide a drop-in replacement, although it is somewhat wierd (e.g. env_parse_enums is always True).

Testing

100% test coverage is provided for latest Python and pydantic-settings version. Tests are run for all minor pydantic-settings v2 versions and all minor Python 3 versions supported by them:

• pyXY — Python 3.{8,9,10,11,12,13}
• psXY — pydantic-settings v2.{2,3,4,5,6,7,8}

	ps29	ps28	ps27	ps26	ps25	ps24	ps23	ps22
py313	✳️	✳️	✅	✅	✅	☑️	✳️	☑️
py312	✅	✅	✅	✅	✅	☑️	☑️	☑️
py311	✅	✅	✅	✅	✅	☑️	☑️	☑️
py310	✅	✅	✅	✅	✅	☑️	☑️	☑️
py39	✅	✅	✅	✅	✅	☑️	☑️	☑️
py38	✅	✅	✅	✅	✅	☑️	☑️	☑️

• ✳️ pytest and mypy passing, coverage report generated
• ✅ pytest and mypy passing
• ☑️ pytest passing, mypy not attempted
• ❌ tests failing or not attempted

History

• September 2024 — Multiple secrets_dir feature was merged to pydantic-settings v2.5.0

Contributing

Pull requests, feature requests, and bug reports are welcome!

• Contribution guidelines

Authors

• Michael Makukha

See also

• Documentation
• Issues
• Changelog
• Security Policy
• Contribution Guidelines
• Code of Conduct
• License