A collection of useful logging formatters and filters.Logging Context, JSON Formatter, Extra Formatter, ISO Time Filter, Flask Filter, Django Filter, ...
This package implements some useful logging utilities. Here below are the main features of the package:
extraYYYY-MM-DDThh:mm:ss.sss±hh:mmAll features can be fully configured from the configuration file.
NOTE: only python 3 is supported
:warning: Version 3.x.x BREAKING CHANGES see Breaking Changes
logging_utilities is available on PyPI.
Use pip to install:
pip install logging-utilities
New release and publish on PyPI is done automatically upon PR merge into master branch. For bug fixes and small new features,
PR can be directly open on master. Then the PR title define the version bump as follow:
#major => major version is bumped#patch or head branch name starts with bug-|hotfix-|bugfix- => patch version is bumpedEvery contribution to this library is welcome ! So if you find a bug or want to add a new feature everyone is welcome to open an issue or created a Pull Request.
Any contribution must follow the git-flow.
You can quickly setup your environment with the makefile:
make setup
This will create a virtual python environment with all packages required for the development.
Note that for pull request, the code MUST BE with yapf formatted and it also MUST PASS the linter. For this you can use the make targets:
make format
make lint
#or
make format-lint
Any new feature should have its unittest class in order to be tested.
When configuring a log formatter you can provide via print style any log record attribute including extra attributes. However when using extra attribute, if this attribute is then missing (e.g. because the logger did not add that extra)
then the logging would raise a ValueError: Formatting field not found in record: ....
For the standard Formatter you could use the Extra Formatter, but if you have any other Formatter you
can use the global logging_utilities.log_record.set_log_record_ignore_missing_factory() method.
The LogRecordIgnoreMissing factory can be used to avoid ValueError exception when formatting a log message from
a log record that don't have the extra required by the formatter.
For example:
import logging
logging.basicConfig(format="%(message)s - %(extra_param)s", level=logging.INFO, force=True)
logger = logging.getLogger('my-logger')
logger.info('My message', extra={'extra_param': 20})
My message - 20
logger.info('My second message')
--- Logging error ---
Traceback (most recent call last):
File "/usr/lib/python3.8/logging/__init__.py", line 440, in format
return self._format(record)
File "/usr/lib/python3.8/logging/__init__.py", line 436, in _format
return self._fmt % record.__dict__
KeyError: 'extra_param'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit
msg = self.format(record)
File "/usr/lib/python3.8/logging/__init__.py", line 929, in format
return fmt.format(record)
File "/usr/lib/python3.8/logging/__init__.py", line 671, in format
s = self.formatMessage(record)
File "/usr/lib/python3.8/logging/__init__.py", line 640, in formatMessage
return self._style.format(record)
File "/usr/lib/python3.8/logging/__init__.py", line 442, in format
raise ValueError('Formatting field not found in record: %s' % e)
ValueError: Formatting field not found in record: 'extra_param'
...
To avoid such crash you can use LogRecordIgnoreMissing that will replace missing extra attributes by an empty string in the message.
import logging
from logging_utilities.log_record import LogRecordIgnoreMissing
logging.setLogRecordFactory(LogRecordIgnoreMissing)
logging.basicConfig(format="%(message)s - %(extra_param)s", level=logging.INFO, force=True)
logger = logging.getLogger('my-logger')
logger.info('My message', extra={'extra_param': 20})
My message - 20
logger.info('My second message')
My second message -
You can also change the default value by using the helper set_log_record_ignore_missing_factory()
import logging
from logging_utilities.log_record import set_log_record_ignore_missing_factory
set_log_record_ignore_missing_factory('my-default')
logging.basicConfig(format="%(message)s - %(extra_param)s", level=logging.INFO, force=True)
logger = logging.getLogger('my-logger')
logger.info('My message', extra={'extra_param': 20})
My message - 20
logger.info('My second message')
My second message - my-default
:warning: NOTE that setting the log record factory is a global action that affects every logger and formatter
With set_logging_context() you can add a thread based context to every log record. This can be quite usefull if
you want to globally set a context to every log record, for example a Request context in a Pyramid/Django application.
In a Pyramid application it is quite usefull to add to every log record the Request context. This can be done as follow:
# module my_app.logging_tweens
from logging_utilities.context import set_logging_context
def logging_context_tween(handler, registry):
def _logging_context_tween(request):
set_logging_context({
"request": {
"method": request.method,
"path": request.path,
"headers": dict(request.headers)
}
})
return handler(request)
return _logging_context_tween
# MAIN
import logging
from wsgiref.simple_server import make_server
from pyramid.config import Configurator
from pyramid.response import Response
logging.basicConfig(format="%(message)s - %(context)s")
logger = logging.getLogger(__name__)
def hello_world(request):
logger.debug('Request for hello world')
return Response('Hello World!')
if __name__ == '__main__':
with Configurator() as config:
# Register the tween
config.add_tween('my_app.logging_tweens.logging_context_tween')
# Configure the route and view
config.add_route('hello', '/')
config.add_view(hello_world, route_name='hello')
app = config.make_wsgi_app()
server = make_server('0.0.0.0', 6543, app)
server.serve_forever()
# A GET / request would produce the following log
'Request for hello world - {"request": {"method": "GET", "path": "/", "headers": {}}}'
For more information on Pyramid Tweens see Registering Tween
JsonFormatter is a python logging formatter that transform the log output into a json object.
JSON log format is quite useful especially when the logs are sent to LogStash.
This formatter supports embedded object as well as array.
The format can be configured either using the format config parameter or the fmt constructor parameter. This parameter should be a dictionary (for Python version below 3.7, it is better to use OrderedDict to keep the attribute order). Each key is taken as such as key for the output JSON object, while each value is transformed as follow in the output:
| Value | Type | Transformation | Example |
|---|---|---|---|
LogRecord attribute | string | The string is a LogRecord attribute name,then the value of this attribute is used as output. See also Type Consistency. | "message" |
LogRecord attribute dotted key | string | The string is a dotted key to access a sub key of a LogRecord dictionary attribute.For example if the LogRecord contains a dictionary attribute added via an extra, you can use the dotted notation to access only a sub object/value of this dictionary. Note if the dotted key attribute doesn't exists it will raise a ValueError unless you set ignore_missing=True in the Formatter config. In the latest case missing attribute will be replaced by '' unless the dotted key has a trailing . then the default value will be {} instead of ''.See also Type Consistency. | "request.path" |
| Named string format | string | The string contains named string format, each named format are replaced by the corresponding LogRecord attribute value. When using the % string formatting style, you can also used dotted notation to access dictionary sub-key; %(request.headers)s. NOTE that in string format the dictionary key must be a valid python attribute name (cannot contain spaces or special characters). | "%(asctime)s.%(msecs)s" |
| Object | dict | The object is embedded in the output with its value following the same rules as defined in this table. | {"lineno": "lineno", "file": "filename", "id": "%(process)x/%(thread)x", "message": "message"} |
| Array | list | The list is embedded as an array in the output. Each value is processed using the rules from this table | ["created", "asctime", "message", "%(process)x/%(thread)x"] |
:warning: If the value doesn't match any of the table above it will raise a ValueError unless you specify ignore_missing=True in the configuration
You can find the LogRecord attributes list in Python Doc
See below the Basic Usage for more examples.
You can change some behavior using the JsonFormatter constructor:
| Parameter | Type | Default | Description |
|---|---|---|---|
fmt | dict | {'levelname': 'levelname', 'name': 'name', 'message': 'message'} | Define the output format, see Configure JSON Format |
datefmt | string | None | Date format for asctime, see time.strftime() |
style | string | % | String formatting style, see logging.Formatter |
add_always_extra | bool | False | When True, logging extra (logging.log('message', extra={'my-extra': 'some value'})) are always added to the output. Otherwise they are only added if present in fmt. |
filter_attributes | list | None | When the formatter is used with a Logging.Filter that adds LogRecord attributes, they can be listed here to avoid to be treated as logging extra. |
remove_empty | bool | False | When True, empty values (empty list, dict, None or empty string) are removed from output. |
ignore_missing | bool | False | If True, then all extra attributes from the log record that are missing (accessed by the fmt parameter) will be replaced by an empty string instead of raising a ValueError exception. NOTE: This has an impact on all formater not only on this one, see LogRecordIgnoreMissing. |
The constructor parameters can be also be specified in the log configuration file using the () class specifier instead of class:
formatters:
json:
(): logging_utilities.formatters.json_formatter.JsonFormatter
add_always_extra: True
fmt:
time: asctime
level: levelname
logger: name
module: module
message: message
:warning: When using the INI file format like documented here, you cannot use the JSON formatter options describe above and have to use the formatter using the class, format, datefmt and style attributes like below
[formatters]
keys = my_json
[formatter_my_json]
class = logging_utilities.formatters.json_formatter.JsonFormatter
format: {
"time": "asctime",
"level": "levelname",
"logger": "name",
"module": "module",
"function": "funcName",
"pid_tid": "%(process)x/%(thread)x",
"message": "message",
"exc_info": "exc_info"
} # OPTIONAL
datefmt = %Y-%m-%d %H:%M # OPTIONAL
style = % # OPTIONAL
When you use ignore_missing=True, all missing attributes from the log record will be replaced by an empty string. This can be an issue if you require type consistency accross JSON logs. To avoid this, you can use the trailing dot notation.
| Single trailing dot | attribute_name. | Default to {} when attribute_name is missing from log record |
| Double trailing dot | attribute_name.. | Default to [] when attribute_name is missing from log record |
This is quite usefull if you want to add a list or an object in your JSON from a LogRecord that might be missing. For example when using the Flask Request Context and you want to add the headers dictionary as object, you can do as follow:
fmt={"message": "message", "request": {"headers": "flask_request_headers."}}
This way if the log record is outside a Flask request, your log output would be
{"message": "this is the message", "request": {"headers": {}}}
instead of
{"message": "this is the message", "request": {"headers": ""}}
and when the record is within a Flask context you will have
{"message": "this is the message", "request": {"headers": {"Host": "www.example.com", ...}}}
This formatter enhance the python standard formatter to allow working with the log extra.
When adding an extra keyword in the format, the python standard formatter raises a ValueError()
when this keyword is missing from log record. This means that if you want to display a log
extra, you have to make sure that every log message contains this extra.
This formatter allow you to provide an extra_fmt parameter that will add record extra to the
log message when available. You can either add the entire extra dictionary: extra_fmt='%s' or
only some extras: extra_fmt='%(extra1)s:%(extra2)s'. In the latest case, when a key is missing
in extra, the value is replaced by extra_default.
When using the whole extra dictionary, you can use extra_pretty_print to improve the
formatting, note that in this case the log might be on multiline (this use pprint.pformat).
See logging.Logger.debug for more infos on the logging extra
Support the same arguments as the logging.Formatter plus the followings:
| Parameter | Type | Default | Description |
|---|---|---|---|
| extra_fmt | None|str | None | When not None, adds the extra at the end of the log message. Either uses named placeholder with the extra keywords or add the whole extra directory using %s. |
| extra_default | None|str | '' | When extra_fmt contains named placeholders and one or more of these placeholders are not found in the log record, then the formatter uses this default value instead. |
| extra_default | any | '' | When using extra_fmt with named placeholders and a keyword is missing in the log record, it is then replaced by this value. |
| extra_pretty_print | boolean | False | When extra_fmt='%s' you can set this flag to True to use pprint.pformat on the dictionary. |
| pretty_print_kwargs | None|dict | None | kwargs as dictionary to pass to pprint.pformat |
formatters:
standard:
(): logging_utilities.formatters.extra_formatter.ExtraFormatter
format: "%(levelname)s - %(name)s - %(message)s"
extra_fmt: " - extra:\n%s"
extra_pretty_print: True
NOTE: ExtraFormatter only support the special key '()' factory in the configuration file (it doesn't work with the normal 'class' key).
When using logging within a Flask application, you can use this Filter to add some context attributes to all LogRecord.
All Flask Request attributes are supported and they are added as LogRecord with the flask_request_ prefix. See Flask Request for more details on available attributes.
| Parameter | Type | Default | Description |
|---|---|---|---|
| attributes | list | None | List of Flask Request attributes name to add to the LogRecord |
version: 1
root:
handlers:
- console
level: DEBUG
propagate: True
filters:
flask:
(): logging_utilities.filters.flask_attribute.FlaskRequestAttribute
attributes:
- url
- method
- headers
- json
formatters:
console:
format: "%(asctime)s - %(message)s - %(flask_request_url)s %(flask_request_method)s %(flask_request_headers)s: %(flask_request_json)s"
handlers:
console:
class: logging.StreamHandler
formatter: console
stream: ext://sys.stdout
filters:
- flask
NOTE: FlaskRequestAttribute only support the special key '()' factory in the configuration file (it doesn't work with the normal 'class' key).
If you want to log the Django HttpRequest object using the JSON Formatter, this filter is for made for you. It converts the record.http_request attribute (or the attribute specified by attr_key in the constructor) to a valid json object if it is of type HttpRequest.
The HttpRequest attributes that are converted can be configured using the include_keys and/or exclude_keys filter parameters. This can be useful if you want to limit the log data, for example if you don't want to log Authentication headers.
:warning: The django framework adds sometimes an HttpRequest or socket object under record.request when
logging. So if you decide to use the attribute name request for this filter, beware that you
will need to handle the case where the attribute is of type socket separately, for example by
filtering it out using the attribute type filter. (see example Filter out LogRecord attributes based on their types)
Add the filter to the log handler and then add simply the HttpRequest to the log extra as follow:
logger.info('My message', extra={'http_request': request})
| Parameter | Type | Default | Description |
|---|---|---|---|
include_keys | list | None | All request attributes that match any of the dotted keys of the list will be added to the jsonifiable object. When None then all attributes are added (default behavior). |
exclude_keys | list | None | All request attributes that match any of the dotted keys of the list will not be added to the jsonifiable object. NOTE this has precedence to include_keys which means that if a key is in both lists, then it is not added. |
attr_key | str | http_request | The name of the attribute that stores the HttpRequest object. It will be replaced in place by a jsonifiable dict representing this object. (Note that django sometimes stores an HttpRequest under attr_key: request. This is however not the default as django also stores other types of objects under this attribute name.) |
filters:
django:
(): logging_utilities.filters.django_request.JsonDjangoRequest
attr_key: 'http_request' # This is the default, so it can be omitted
include_keys:
- http_request.META.REQUEST_METHOD
- http_request.META.SERVER_NAME
- http_request.environ
exclude_keys:
- http_request.META.SERVER_NAME
- http_request.environ.wsgi
NOTE: JsonDjangoRequest only support the special key '()' factory in the configuration file (it doesn't work with the normal 'class' key).
If different libraries or different parts of your code log different object types under the same logRecord extra attribute, you can use this filter to keep only some of them (whitelist mode) or filter out some of them (blacklist mode).
| Parameter | Type | Default | Description |
|---|---|---|---|
typecheck_list | dict(key, type|list of types) | None | A dictionary that maps keys to a type or a list of types. By default, it will only keep a parameter matching a key if the types match or if any of the types in the list match (white list). If in black list mode, it will only keep a parameter if the types don't match. Parameters not appearing in the dict will be ignored and passed though regardless of the mode (whitelist or blacklist). |
is_blacklist | bool | false | Whether the list passed should be a blacklist or a whitelist. To use both, simply include this filter two times, one time with this parameter set true and one time with this parameter set false. |
filters:
type_filter:
(): logging_utilities.filters.attr_type_filter.AttrTypeFilter
is_blacklist: False # Default value is false, so this could be left out
typecheck_list:
# For each attribute listed, one type or a list of types can be specified
request: # can only be a toplevel attribute (no dotted keys allowed)
- django.http.request.HttpRequest # Can be a class name only or the full dotted path
- builtins.dict
my_attr: myClass
The standard logging doesn't support the time as ISO with timezone; YYYY-MM-DDThh:mm:ss.sss±hh:mm. By default asctime uses a ISO like format; YYYY-MM-DD hh:mm:ss.sss, but without T separator (although this one could be configured by overriding a global variable, this can't be done by config file). You can use the datefmt option to specify another date format, however this one don't supports milliseconds, so you could achieve this format: YYYY-MM-DDThh:mm:ss±hh:mm.
This Filter can be used to achieve the full ISO 8601 Time format including timezone and milliseconds.
| Parameter | Type | Default | Description |
|---|---|---|---|
isotime | bool | True | Add log local time as isotime attribute to LogRecord with the YYYY-MM-DDThh:mm:ss.sss±hh:mm format. |
utc_isotime | bool | False | Add log UTC time as utc_isotime attribute to LogRecord with the YYYY-MM-DDThh:mm:ss.sss±hh:mm format. |
filters:
isotime:
(): logging_utilities.filters.TimeAttribute
utc_isotime: True
isotime: False
NOTE: TimeAttribute only support the special key '()' factory in the configuration file (it doesn't work with the normal 'class' key).
Simple logging Filter to add constant attribute to every LogRecord
filters:
application:
(): logging_utilities.filters.ConstAttribute
application: my-application
NOTE: ConstAttribute only support the special key '()' factory in the configuration file (it doesn't work with the normal 'class' key).
Sometimes you might want to have different log Level based on the logger and handler. The standard logging library allow to set a logger level or a handler level but not based on both. Let say you have a config with two loggers logging to two handlers, on the first handler you want all messages of both loggers and on the second handler you want all messages of the first logger but only the WARNING messages of the second logger. This is here were this filter come into play.
| Parameter | Type | Default | Description |
|---|---|---|---|
| level | int | string | 'DEBUG' | All messages with a lower level than this one will be filtered out. |
| logger | string | '' | When non empty, only message from this logger will be filtered out based on their level. |
root:
handlers:
- "console"
- "file"
level: "DEBUG"
propagate: "True"
filters:
B_filter:
(): logging_utilities.filters.LevelFilter
level: "WARNING"
logger: 'B'
loggers:
A:
level: "DEBUG"
B:
level: "DEBUG"
handlers:
console:
class: "logging.StreamHandler"
file:
class: "logging.handlers.RotatingFileHandler"
filters:
- "B_filter"
NOTE: LevelFilter only support the special key '()' factory in the configuration file (it doesn't work with the normal 'class' key).
AddToThreadContextMiddleware is a Middleware with which you can add the Django HttpRequest to thread local variables. The request object is added to a global variable in logging_utilities.thread_context and can be accessed in the following way:
from logging_utilities.thread_context import thread_context
getattr(thread_context, 'request')
AddThreadContextFilter provides a logging filter that will add data from the thread local store logging_utilities.thread_context to the log record. To set data on the thread store do the following:
from logging_utilities.thread_context import thread_context
setattr(thread_context, 'key', data)
Configure the filter to decide which data should be added and how it should be named:
filters:
add_request:
(): logging_utilities.filters.add_thread_context_filter.AddThreadContextFilter
contexts:
- logger_key: log_record_key
context_key: key
| Parameter | Type | Default | Description |
|---|---|---|---|
contexts | list | empty | List of values to add to the log record. Dictionary must contain value for 'context_key' to read value from thread local variable. Dictionary must also contain 'logger_key' to set the value on the log record. |
import logging
from logging_utilities.formatters.json_formatter import basic_config
# default keyword parameter `format`: """{"levelname": "levelname", "name": "name", "message": "message"}"""
basic_config(level=logging.INFO)
logging.info('hello, json_formatter')
output:
{"levelname": "INFO", "name": "root", "message": "hello, json_formatter"}
import logging
from logging_utilities.formatters.json_formatter import JsonFormatter
# `FORMAT` can be `json`, `OrderedDict` or `dict`.
# If `FORMAT` is `dict` and python version < 3.7.0, the output order is sorted by keys, otherwise it will be the same
# as the defined order.
#
# KEY := string, can be whatever you like.
# VALUE := `LogRecord` attribute name, string, formatted string (e.g. "%(asctime)s.%(msecs)s"), list or dict
FORMAT = {
"Name": "name",
"Levelno": "levelno",
"Levelname": "levelname",
"Pathname": "pathname",
"Filename": "filename",
"Module": "module",
"Lineno": "lineno",
"FuncName": "funcName",
"Created": "created",
"Asctime": "asctime",
"Msecs": "msecs",
"RelativeCreated": "relativeCreated",
"Thread": "thread",
"ThreadName": "threadName",
"Process": "process",
"Message": "message"
}
root = logging.getLogger()
root.setLevel(logging.INFO)
formatter = JsonFormatter(FORMAT)
sh = logging.StreamHandler()
sh.setFormatter(formatter)
sh.setLevel(logging.INFO)
root.addHandler(sh)
def test():
root.info("test %s format", 'string')
test()
output:
{
"Name": "root",
"Levelno": 20,
"Levelname": "INFO",
"Pathname": "test.py",
"Filename": "test.py",
"Module": "test",
"Lineno": 75,
"FuncName": "test",
"Created": 1588185267.3198836,
"Asctime": "2020-04-30 02:34:27,319",
"Msecs": 319.8835849761963,
"RelativeCreated": 88.2880687713623,
"Thread": 16468,
"ThreadName": "MainThread",
"Process": 16828,
"Message": "test string format"
}
config.yaml:
version: 1
root:
handlers:
- console
level: DEBUG
propagate: True
formatters:
json:
class: logging_utilities.formatters.json_formatter.JsonFormatter
format:
time: asctime
level: levelname
logger: name
module: module
function: funcName
process: process
thread: thread
message: message
handlers:
console:
class: logging.StreamHandler
formatter: json
stream: ext://sys.stdout
Then in your python code use it as follow:
import logging
import logging.config
import yaml
config = {}
with open('example-config.yaml', 'r') as fd:
config = yaml.safe_load(fd.read())
logging.config.dictConfig(config)
root = logging.getLogger()
root.info('Test file config')
output:
{
"function": "<module>",
"level": "INFO",
"logger": "root",
"message": "Test file config",
"module": "<stdin>",
"process": 12264,
"thread": 139815989413696,
"time": "asctime"
}
config.yaml
version: 1
root:
handlers:
- console
level: DEBUG
propagate: True
filters:
isotime:
(): logging_utilities.filters.TimeAttribute
flask:
(): logging_utilities.filters.flask_attribute.FlaskRequestAttribute
attributes:
- url
- method
- headers
- remote_addr
- json
formatters:
json:
class: logging_utilities.formatters.json_formatter.JsonFormatter
format:
time: isotime
level: levelname
logger: name
module: module
function: funcName
process: process
thread: thread
request:
# We use the "%()s" notation here to ensure a string output and also if the LogRecord has
# no flask context, meaning no `flask_request_url` attribute, the "%()s" notation ensure
# to have an empty string instead of treating `flask_request_url` as a string constant.
url: "%(flask_request_url)s"
method: "%(flask_request_method)s"
# We use a trailing dot here to ensure to have a dictionary output even if the LogRecord
# doesn't have a flask_request_headers attribute.
headers: flask_request_headers.
data: flask_request_json.
remote: "%(flask_request_remote_addr)s"
message: message
handlers:
console:
class: logging.StreamHandler
formatter: json
stream: ext://sys.stdout
filters:
- isotime
- flask
NOTE: This require to have flask package installed otherwise it raises ImportError
Then in your python code use it as follow:
import logging
import logging.config
import yaml
from flask import Flask
config = {}
with open('example-config.yaml', 'r') as fd:
config = yaml.safe_load(fd.read())
logging.config.dictConfig(config)
app = Flask('test')
root = logging.getLogger()
with app.test_request_context("path/test", method='GET', headers={"Accept": "*/*"}):
root.info('Test file config')
output:
{
"time": "2022-07-20T10:09:10.765237+02:00",
"level": "INFO",
"logger": "root",
"module": "<stdin>",
"function": "<module>",
"process": 58043,
"thread": 139717802334016,
"request": {
"url": "http://localhost/path/test",
"method": "GET",
"headers": {
"Host": "localhost",
"Accept": "*/*"
},
"data": null,
"remote": null
},
"message": "Test file config"
}
config.yaml
version: 1
root:
handlers:
- console
level: DEBUG
propagate: True
filters:
isotime:
(): logging_utilities.filters.TimeAttribute
django:
(): logging_utilities.filters.django_request.JsonDjangoRequest
include_keys:
- http_request.path
- http_request.method
- http_request.headers
exclude_keys:
- http_request.headers.Authorization
- http_request.headers.Proxy-Authorization
formatters:
json:
class: logging_utilities.formatters.json_formatter.JsonFormatter
format:
time: isotime
level: levelname
logger: name
module: module
function: funcName
process: process
thread: thread
request: http_request
response: response
message: message
handlers:
console:
class: logging.StreamHandler
formatter: json
stream: ext://sys.stdout
filters:
- isotime
- django
NOTE: This require to have django package installed otherwise it raises ImportError
Then in your python code use it as follow:
import logging
import logging.config
import yaml
from django.http import JsonResponse
from django.conf import settings
from django.test import RequestFactory
config = {}
with open('example-config.yaml', 'r') as fd:
config = yaml.safe_load(fd.read())
logging.config.dictConfig(config)
logger = logging.getLogger('your_logger')
def my_page(request):
answer = {'success': True}
logger.info('My page requested', extra={'request': request, 'response': answer})
return JsonResponse(answer)
settings.configure()
factory = RequestFactory()
my_page(factory.get('/my_page?test=true'))
output:
{
"function": "my_page",
"level": "INFO",
"logger": "your_logger",
"message": "My page requested",
"module": "<stdin>",
"process": 20421,
"request": {
"method": "GET",
"path": "/my_page",
"headers": {
"Cookie": ""
}
},
"response": {
"success": true
},
"thread": 140433370822464,
"time": "2020-10-12T16:44:45.374508+02:00"
}
Let's say you want to log parts of the django HttpRequest in Json format. Django already logs it
sometimes under record.request so you can use the django request filter to transform it to a jsonisable
dictionary. However django sometimes also logs an object of type socket.socket that you may not
want to include in the logs. In this case you could use the following configuration. This will only
keep the request attribute if it is of type HttpRequest.
config.yaml
version: 1
root:
handlers:
- console
level: DEBUG
propagate: True
filters:
type_filter:
(): logging_utilities.filters.attr_type_filter.AttrTypeFilter
typecheck_list:
request: django.http.request.HttpRequest
isotime:
(): logging_utilities.filters.TimeAttribute
django:
(): logging_utilities.filters.django_request.JsonDjangoRequest
attr_name: request
include_keys:
- request.path
- request.method
- request.headers
formatters:
json:
class: logging_utilities.formatters.json_formatter.JsonFormatter
format:
time: isotime
level: levelname
logger: name
module: module
function: funcName
process: process
thread: thread
request_path: request.path
request_method: request.method
request:
# NOTE: django headers name are case sensitive
header.accept: request.headers.Accept
header.accept-encoding: request.headers.Accept-Encoding
header.accept_language: request.headers.Accept-Language
message: message
handlers:
console:
class: logging.StreamHandler
formatter: json
stream: ext://sys.stdout
filters:
- isotime
# Typefilter must be before django filter, as the django filter
# will modify the type of the "HttpRequest" object
- type_filter
- django
NOTE: This require to have django package installed otherwise it raises ImportError
Then in your python code use it as follow:
import logging
import logging.config
import yaml
from django.http import JsonResponse
from django.conf import settings
from django.test import RequestFactory
config = {}
with open('example-config.yaml', 'r') as fd:
config = yaml.safe_load(fd.read())
logging.config.dictConfig(config)
logger = logging.getLogger('your_logger')
def my_page(request):
answer = {'success': True}
logger.info('My page requested', extra={'request': request})
return JsonResponse(answer)
settings.configure()
factory = RequestFactory()
my_page(factory.get(
'/my_page?test=true',
HTTP_ACCEPT='*/*',
HTTP_ACCEPT_ENCODING='gzip',
HTTP_ACCEPT_LANGUAGE='en')
)
output:
{
"time": "2022-07-20T12:29:19.536922+02:00",
"level": "INFO",
"logger": "your_logger",
"module": "<stdin>",
"function": "my_page",
"process": 78479,
"thread": 139751209555776,
"request_path": "/my_page",
"request_method": "GET",
"request": {
"header.accept": "*/*",
"header.accept-encoding": "gzip",
"header.accept_language": "en"
},
"message": "My page requested"
}
config.yaml
version: 1
root:
handlers:
- console
level: DEBUG
propagate: True
filters:
type_filter:
(): logging_utilities.filters.attr_type_filter.AttrTypeFilter
typecheck_list:
request: django.http.request.HttpRequest
isotime:
(): logging_utilities.filters.TimeAttribute
django:
(): logging_utilities.filters.django_request.JsonDjangoRequest
attr_name: request
include_keys:
- request.path
- request.method
- request.headers
exclude_keys:
- request.headers.Authorization
- request.headers.Proxy-Authorization
formatters:
standard_extra:
(): logging_utilities.formatters.extra_formatter.ExtraFormatter
# NOTE also in the constructor the parameter is `fmt` we need to use `format` here
format: "%(isotime)s - %(levelname)s - %(name)s - %(message)s"
extra_fmt: " - extra:\n%s"
extra_pretty_print: True
pretty_print_kwargs:
indent: 2
width: 60
handlers:
console:
class: logging.StreamHandler
formatter: standard_extra
stream: ext://sys.stdout
filters:
- isotime
# Type filter must be before django filter
- type_filter
- django
NOTE: This require to have django package installed otherwise it raises ImportError
Then in your python code use it as follow:
#!.venv/bin/python3
import logging
import logging.config
import yaml
from django.http import JsonResponse
from django.conf import settings
from django.test import RequestFactory
config = {}
with open('example-config.yaml', 'r') as fd:
config = yaml.safe_load(fd.read())
logging.config.dictConfig(config)
logger = logging.getLogger('your_logger')
def my_page(request):
answer = {'success': True}
logger.info('My page requested', extra={'request': request, 'response': answer})
return JsonResponse(answer)
settings.configure()
factory = RequestFactory()
my_page(factory.get('/my_page?test=true'))
output:
2020-11-19T13:32:58.942568+01:00 - INFO - your_logger - My page requested - extra:
{ 'request': { 'headers': {'Cookie': ''},
'method': 'GET',
'path': '/my_page'},
'response': {'success': True}}
config.yaml
version: 1
root:
handlers:
- console
level: DEBUG
propagate: True
formatters:
standard_extra:
(): logging_utilities.formatters.extra_formatter.ExtraFormatter
# NOTE also in the constructor the parameter is `fmt` we need to use `format` here
format: "%(asctime)s - %(levelname)s - %(name)s - %(message)s"
extra_fmt: " - extra1=%(extra1)s"
handlers:
console:
class: logging.StreamHandler
formatter: standard_extra
stream: ext://sys.stdout
Then in your python code use it as follow:
#!.venv/bin/python3
import logging
import logging.config
import yaml
config = {}
with open('example-config.yaml', 'r') as fd:
config = yaml.safe_load(fd.read())
logging.config.dictConfig(config)
logger = logging.getLogger('your_logger')
logger.debug('My log with extras', extra={'extra1': 23, 'extra2': "don't add this"})
output:
2020-11-19 13:42:29,424 - DEBUG - your_logger - My log with extras - extra1=23
Combine the use of the middleware AddToThreadContextMiddleware with the filters AddThreadContextFilter and JsonDjangoRequest, as well as the JsonFormatter to add request context to each log entry.
Activate the middleware:
MIDDLEWARE = (
...,
'logging_utilities.django_middlewares.add_request_context.AddToThreadContextMiddleware',
...,
)
Configure the filters AddThreadContextFilter and JsonDjangoRequest to add the request from the thread variable to the log record and make it json encodable. Use the JsonFormatter to format the request values
filters:
add_request:
(): logging_utilities.filters.add_thread_context_filter.AddThreadContextFilter
contexts:
- context_key: request # Must be value 'request' as this is how the middleware adds the value.
logger_key: request
request_fields:
(): logging_utilities.filters.django_request.JsonDjangoRequest
attr_key: request # Must match the above logger_key
include_keys:
- request.path
- request.method
formatters:
json:
(): logging_utilities.formatters.json_formatter.JsonFormatter
fmt:
time: asctime
level: levelname
logger: name
module: module
message: message
request:
path: request.path
method: request.method
handlers:
console:
formatter: json
filters:
# Make sure to add the filters in the correct order.
# These filters modify the record in-place, and as the record is passed serially to each handler.
- add_request
- request_fields
From version 3.x.x to version 4.x.x there is the following breaking change:
record.http_request instead of
the attribute record.request. There is however a new option attr_name in the filters constructor
to manually specify the attribute name. See the example Add parts of Django Request to JSON Output for an example on how to use attr_name to be
backward-compatible with 3.x.xFrom version 2.x.x to version 3.x.x there is the following breaking change:
fmt parameter. Now if you want to have a string constant in all of you JSON logs output, you need to use the Constant Record Attribute Filter.From version 1.x.x to version 2.x.x there is the following breaking change:
ValueError exception due to the missing Flask Request attribute. To avoid this you can use the new LogRecordIgnoreMissing.The JSON Formatter implementation has been inspired by MyColorfulDays/jsonformatter
The Request Var middleware has been inspired by kindlycat/django-request-vars
FAQs
A collection of useful logging formatters and filters.Logging Context, JSON Formatter, Extra Formatter, ISO Time Filter, Flask Filter, Django Filter, ...
We found that logging-utilities demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 3 open source maintainers collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
In this segment of the Risky Business podcast, Feross Aboukhadijeh and Patrick Gray discuss the challenges of tracking malware discovered in open source softare.
Research
Security News
A threat actor's playbook for exploiting the npm ecosystem was exposed on the dark web, detailing how to build a blockchain-powered botnet.
Security News
NVD’s backlog surpasses 20,000 CVEs as analysis slows and NIST announces new system updates to address ongoing delays.