powerflex-logging-utilities
Helpful code for logging in Python by PowerFlex.
Module | Description |
---|
forbid_toplevel_logging | Disable logging with the top-level root logging functions such as logging.info . |
log_slow_callbacks | Either warn or info log when an async callback runs for too long. |
init_loggers | A function for easily setting up logging to a file and to stdout. |
Class | Description |
---|
JsonFormatter | A JSON log formatter to enable structured logging. It depends on the python-json-logger package. |
TraceLogger | A Python Logger subclass that adds a TRACE logging level |
AsyncNatsLogLevelListener | A NATS interface for changing the program's log level by sending a NATS request |
Installation
You can install from PyPi directly:
pip install powerflex-logging-utilities
Sample usage
Initializing Loggers
Setup all Loggers to output JSON to stdout and to a file:
import logging
import sys
from powerflex_logging_utilities import (
JsonFormatter,
init_loggers,
TraceLogger,
)
LOG_LEVEL = "DEBUG"
FILE_LOG_LEVEL = "TRACE"
LOG_FILE = "./logs/trace.log"
MAX_LOG_FILE_MB = 10
MAX_TOTAL_LOG_FILE_MB = 10000
root_logger = logging.getLogger()
logging.captureWarnings(True)
logging.getLogger("parso").propagate=False
init_loggers.init_loggers(
[root_logger],
log_level=LOG_LEVEL,
file_log_level=FILE_LOG_LEVEL,
filename=LOG_FILE,
max_bytes=1000 * 1000 * MAX_LOG_FILE_MB,
backup_count=MAX_TOTAL_LOG_FILE_MB // MAX_LOG_FILE_MB,
stream=sys.stdout,
formatter=JsonFormatter,
info_logger=root_logger,
)
logging.setLoggerClass(TraceLogger)
logger = logging.getLogger(__name__)
This uses Python's logger propagation feature.
We only need to configure the root Logger in order to make sure all other Loggers output in the desired format.
You can pass formatter_kwargs
to enable logging with a different JSON serializer.
To use:
logger = logging.getLogger(__name__)
logger.info("hello world")
Explicitly listing loggers
You can also list the loggers you'd like to configure instead of configuring
the root logger.
This could be useful if you configure your package's main logger
logging.getLogger("package")
. You can then use Python's logger propagation by calling
logging.getLogger("package.submodule.a.b.c")
to get Logger instances for all
other submodules.
import logging
from powerflex_logging_utilities import (
JsonFormatter,
init_loggers,
)
logger = logging.getLogger("your_package_name")
logging.captureWarnings(True)
init_loggers.init_loggers(
[logger, "asyncio", "py.warnings"],
log_level="DEBUG",
file_log_level="TRACE",
filename="./logs/trace-no-root.log",
formatter=JsonFormatter,
info_logger=logger,
)
NOTICE: if you use this method, any loggers you do not explicitly list will have non-JSON output.
Using several other utilities
import logging
from powerflex_logging_utilities import (
forbid_toplevel_logging,
log_slow_callbacks,
)
logger = logging.getLogger(__name__)
log_slow_callbacks.log_slow_callbacks(logger)
forbid_toplevel_logging.forbid_logging_with_logging_toplevel()
Using the JSON formatter
import logging
import sys
from powerflex_logging_utilities import JsonFormatter
log_handler = logging.StreamHandler(stream=sys.stdout)
log_handler.setLevel("DEBUG")
log_handler.setFormatter(JsonFormatter())
logger = logging.getLogger(__name__)
logger.addHandler(log_handler)
logger.setLevel("DEBUG")
logger.info("hello world", extra={
"data": ["log structured data", ":D"],
1: "handles non string key",
})
{
"message": "hello world",
"name": "__main__",
"module": "<ipython-input-10-b016ce80d46f>",
"lineno": 1,
"funcName": "<cell line: 1>",
"filename": "<ipython-input-10-b016ce80d46f>",
"asctime": "2022-05-12 01:04:16,824",
"data": [
"log structured data",
":D"
],
"severity": "INFO",
"1": "handles non string key"
}
Using pipenv
- Run
make setup-with-pipenv
to install all dependencies.
Make sure you have the version of Python specified in .tool-versions
or simply change this file to your Python version (must be 3.8+). - Run
pipenv shell
or run the following make
commands with pipenv run make ...
.
You could also alias pmake
to pipenv run make
for convenience.
Tests
There is 100% code coverage.
make test-unit
To test in several versions of Python, run:
tox
To download several versions of Python, use pyenv
or asdf
To use pyenv
, install it here and run the following script:
./install_python_versions_pyenv.sh
To use asdf
, install the core parts here and run the following commands:
./install_python_versions_asdf.sh
Testing the code in this README
make test-readme
Checking code quality
The Github Actions will run all of the following checks on the code.
Code formatting
make format-fix
Linting
make lint
Type checking
make type-check-strict
- Make sure all code checks have passed with
make commitready
. - Make sure you commit all code you wish to release with
git commit
. - Set the version in
./src/powerflex_monitoring/VERSION
Please attempt to follow semantic versioning. - Run
make bump-version
to commit the change to the VERSION
file. - Run
make release
to upload the package to pypi.org and to push a new git tag