django-request-profiler

Django Request Profiler

This package now requires Python 3.9 and Django 3.2 and above.

A very simple request profiler for Django.

Introduction

Premature optimization is the root of all evil.

There are a lot of very good, and complete, python and django profilers available. They can give you detailed stack traces and function call timings, output all the SQL statements that have been run, the templates that have been rendered, and the state of any / all variables along the way. These tools are great for optimisation of your application, once you have decided that the time is right.

django-request-profiler is not intended to help you optimise, but to help you decide whether you need to optimise in the first place. It is complimentary.

Requirements

1. Small enough to run in production
2. Able to configure profiling at runtime
3. Configurable to target specific URLs or users
4. Record basic request metadata:

• Duration (request-response)
• Request path, remote addr, user-agent
• Response status code, content length
• View function
• Django user and session keys (if appropriate)
• Database query count (if DEBUG=True)

It doesn't need to record all the inner timing information - the goal is to have a system that can be used to monitor site response times, and to identify problem areas ahead of time.

Technical details

The profiler itself runs as Django middleware, and it simply starts a timer when it first sees the request, and stops the timer when it is finished with the response. It should be installed as the first middleware in MIDDLEWARE_CLASSES in order to record the maximum duration.

It hooks into the process_request method to start the timer, the process_view method to record the view function name, and the process_response method to stop the timer, record all the request information and store the instance.

The profiler is controlled by adding RuleSet instances which are used to filter which requests are profiled. There can be many, overlapping, RuleSets, but if any match, the request is profiled. The RuleSet model defines two core matching methods:

1. uri_regex - in order to profile a subset of the site, you can supply a regex which is used match the incoming request path. If the url matches, the request can be profiled.

2. user_filter_type - there are three choices here - profile all users, profile only authenticated users, and profile authenticated users belonging to a given Group - e.g. create a groups called "profiling" and add anyone you want to profile.

These filter properties are an AND (must pass the uri and user filter), but the rules as a group are an OR - so if a request passes all the filters in any rule, then it's profiled.

These filters are pretty blunt, and there are plenty of use cases where you may want more sophisticated control over the profiling. There are two ways to do this. The first is a setting, REQUEST_PROFILER_GLOBAL_EXCLUDE_FUNC, which is a function that takes a request as the single argument, and must return True or False. If it returns False, the profile is cancelled, irrespective of any rules. The primary use case for this is to exclude common requests that you are not interested in, e.g. from search engine bots, or from Admin users etc. The default for this function is to prevent admin user requests from being profiled.

The second control is via the cancel() method on the ProfilingRecord, which is accessible via the request_profile_complete signal. By hooking in to this signal you can add additional processing, and optionally cancel the profiler. A typical use case for this is to log requests that have exceeded a set request duration threshold. In a high volume environment you may want to, for instance, only profile a random subset of all requests.

.. code:: python

from django.dispatch import receiver
from request_profiler.signals import request_profile_complete

@receiver(request_profiler_complete)
def on_request_profile_complete(sender, **kwargs):
    profiler = kwargs.get('instance')
    if profiler.elapsed > 2:
        # log long-running requests
        # NB please don't use 'print' for real - use logging
        print u"Long-running request warning: %s" % profiler
    else:
        # calling cancel means that it won't be saved to the db
        profiler.cancel()

An additional scenario where you may want to use the signal is to store the profiler records async - say if you are recording every request for a short period, and you don't want to add unnecessary inline database write operations. In this case you can use the stop() method, which will prevent the middleware from saving it directly (it will only save records where profiler.is_running is true, and both cancel and stop set it to false).

.. code:: python

from django.dispatch import receiver
from request_profiler.signals import request_profile_complete

@receiver(request_profiler_complete)
def on_request_profile_complete(sender, **kwargs):
    profiler = kwargs.get('instance')
    # stop the profiler to prevent it from being saved automatically
    profiler.stop()
    assert not profiler.is_running
    # add a job to a queue to perform the save itself
    queue.enqueue(profiler.save)

Installation

For use as the app in Django project, use pip:

.. code:: shell

$ pip install django-request-profiler
# For hacking on the project, pull from Git:
$ git pull [e-mail]:yunojuno/django-request-profiler.git

Tests

The app installer contains a test suite that can be run using the Django test runner:

.. code:: shell

$ pip install -r requirements.txt
$ python manage.py test test_app request_profiler

If you want to test coverage you'll need to add some dependencies:

.. code:: shell

$ pip install coverage django-coverage
$ python manage.py test_coverage test_app request_profiler

The tests also run using tox <https://testrun.org/tox/latest/>_:

.. code:: shell

$ pip install tox
$ tox

Note: To test with a custom user model, you should override the default User model by providing a value for the AUTH_USER_MODEL (in testapp/settings) setting that references a custom model

The tests run on Travis <https://travis-ci.org/yunojuno/django-request-profiler>_ on commits to master.

Usage

Once installed, add the app and middleware to your project's settings file. In order to add the database tables, you should run the migrate command:

.. code:: bash

$ python manage.py migrate request_profiler

NB the middleware must be the first item in MIDDLEWARE_CLASSES.

.. code:: python

INSTALLED_APPS = (
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'request_profiler',
)

MIDDLEWARE_CLASSES = [
    # this package's middleware
    'request_profiler.middleware.ProfilingMiddleware',
    # default django middleware
    'django.middleware.common.CommonMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
]

Configuration

To configure the app, open the admin site, and add a new request profiler 'Rule set'. The default options will result in all non-admin requests being profiled.

Licence

MIT (see LICENCE)