logfmter

Add logfmt structured logging using the stdlib logging module and without changing a single log call.

> logging.warn("user created", extra=user)

at=WARNING msg="user created" first_name=John last_name=Doe age=25

Table of Contents

1. Why
2. Install
3. Usage
   1. Integration
   2. Configuration
   3. Extension
   4. Guides
4. Development
   1. Required Software
   2. Getting Started
   3. Publishing

Why

• enables both human and computer readable logs, recommended as a "best practice" by Splunk
• formats all first and third party logs, you never have to worry about a library using a different logging format
• simple to integrate into any existing application, requires no changes to existing log statements i.e. structlog

Install

$ pip install logfmter

Usage

This package exposes a single Logfmter class that can be integrated into the standard library logging system like any logging.Formatter.

Integration

basicConfig

import logging
from logfmter import Logfmter

handler = logging.StreamHandler()
handler.setFormatter(Logfmter())

logging.basicConfig(handlers=[handler])

logging.error("hello", extra={"alpha": 1}) # at=ERROR msg=hello alpha=1
logging.error({"token": "Hello, World!"}) # at=ERROR token="Hello, World!"

dictConfig

If you are using dictConfig, you need to consider your setting of disable_existing_loggers. It is enabled by default, and causes any third party module loggers to be disabled.

import logging.config

logging.config.dictConfig(
    {
        "version": 1,
        "formatters": {
            "logfmt": {
                "()": "logfmter.Logfmter",
            }
        },
        "handlers": {
            "console": {"class": "logging.StreamHandler", "formatter": "logfmt"}
        },
        "loggers": {"": {"handlers": ["console"], "level": "INFO"}},
    }
)

logging.info("hello", extra={"alpha": 1}) # at=INFO msg=hello alpha=1

Notice, you can configure the Logfmter by providing keyword arguments as dictionary items after "()":

...

    "logfmt": {
        "()": "logfmter.Logfmter",
        "keys": [...],
        "mapping": {...}
    }

...

fileConfig

Using logfmter via fileConfig is not supported, because fileConfig does not support custom formatter initialization. There may be some hacks to make this work in the future. Let me know if you have ideas or really need this.

Configuration

keys

By default, the at=<levelname> key/value will be included in all log messages. These default keys can be overridden using the keys parameter. If the key you want to include in your output is represented by a different attribute on the log record, then you can use the mapping parameter to provide that key/attribute mapping.

Reference the Python logging.LogRecord Documentation for a list of available attributes.

import logging
from logfmter import Logfmter

formatter = Logfmter(keys=["at", "processName"])

handler = logging.StreamHandler()
handler.setFormatter(formatter)

logging.basicConfig(handlers=[handler])

logging.error("hello") # at=ERROR processName=MainProceess msg=hello

mapping

By default, a mapping of {"at": "levelname"} is used to allow the at key to reference the log record's levelname attribute. You can override this parameter to provide your own mappings.

import logging
from logfmter import Logfmter

formatter = Logfmter(
    keys=["at", "process"],
    mapping={"at": "levelname", "process": "processName"}
)

handler = logging.StreamHandler()
handler.setFormatter(formatter)

logging.basicConfig(handlers=[handler])

logging.error("hello") # at=ERROR process=MainProceess msg=hello

datefmt

If you request the asctime attribute (directly or through a mapping), then the date format can be overridden through the datefmt parameter.

import logging
from logfmter import Logfmter

formatter = Logfmter(
    keys=["at", "when"],
    mapping={"at": "levelname", "when": "asctime"},
    datefmt="%Y-%m-%d"
)

handler = logging.StreamHandler()
handler.setFormatter(formatter)

logging.basicConfig(handlers=[handler])

logging.error("hello") # at=ERROR when=2022-04-20 msg=hello

Extension

You can subclass the formatter to change its behavior.

import logging
from logfmter import Logfmter


class CustomLogfmter(Logfmter):
    """
    Provide a custom logfmt formatter which formats
    booleans as "yes" or "no" strings.
    """

    @classmethod
    def format_value(cls, value):
        if isinstance(value, bool):
            return "yes" if value else "no"

	return super().format_value(value)

handler = logging.StreamHandler()
handler.setFormatter(CustomLogfmter())

logging.basicConfig(handlers=[handler])

logging.error({"example": True}) # at=ERROR example=yes

Guides

Default Key/Value Pairs

Instead of providing key/value pairs at each log call, you can override the log record factory to provide defaults:

_record_factory = logging.getLogRecordFactory()

def record_factory(*args, **kwargs):
    record = _record_factory(*args, **kwargs)
    record.trace_id = 123
    return record

logging.setLogRecordFactory(record_factory)

This will cause all logs to have the trace_id=123 pair regardless of including trace_id in keys or manually adding trace_id to the extra parameter or the msg object.

Development

Required Software

If you are using nix & direnv, then your dev environment will be managed automatically. Otherwise, you will need to manually install the following software:

• direnv
• git
• pyenv

Getting Started

Setup

If you are using pyenv, you will need to install the correct versions of python using <runtimes.txt xargs -n 1 pyenv install -s.

$ direnv allow
$ pip install -r requirements/dev.txt
$ pre-commit install
$ pip install -e .

Tests

Run the test suite against the active python environment.

$ pytest

Run the test suite against the active python environment and watch the codebase for any changes.

$ ptw

Run the test suite against all supported python versions.

$ tox

Publishing

Create

1. Update the version number in logfmter/__init__.py.

2. Add an entry in HISTORY.md.

3. Commit the changes, tag the commit, and push the tags:

   $ git commit -am "v<major>.<minor>.<patch>"
   $ git tag v<major>.<minor>.<patch>
   $ git push origin main --tags
   

4. Convert the tag to a release in GitHub with the history entry as the description.

Build

$ python -m build

Upload

$ twine upload dist/*