|pypi-badge| |ci-badge| |codecov-badge| |doc-badge| |pyversions-badge| |license-badge| |status-badge|
EdX utilities for Django Application development.
Note that some utilities may warrant their own repository. A judgement call needs to be made as to whether code properly belongs here or not. Please review with the Architecture Team if you have any questions.
This repository includes shared utilities for:
Cache Utilities_: Includes a RequestCache and a TieredCache.
Django User and Group Utilities_: Includes user and group utilities.
IP Address Utilities_: Utilities for handling request IP addresses.
Logging Utilities_: Includes log filters and an encrypted logging helper.
Monitoring Utilities_: Includes Middleware and utilities for enhanced monitoring.
At this time, supports NewRelic monitoring.
Plugin Infrastructure_: Enables enhanced Django Plugin capabilities.
Security Utilities_: Includes a middleware to add CSP response headers.
Data Generation_: Management command for generating Django data based on model factories.
.. _Cache Utilities: edx_django_utils/cache/README.rst
.. _Django User and Group Utilities: edx_django_utils/user/README.rst
.. _IP Address Utilities: edx_django_utils/ip/README.rst
.. _Logging Utilities: edx_django_utils/logging/README.rst
.. _Monitoring Utilities: edx_django_utils/monitoring/README.rst
.. _Plugin Infrastructure: edx_django_utils/plugins/README.rst
.. _Security Utilities: edx_django_utils/security/README.rst
.. _Data Generation: edx_django_utils/data_generation/README.rst
The full documentation is in the docs directory, and is published to https://edx-django-utils.readthedocs.org.
Please see the Open edX documentation for guidance on Python development <https://docs.openedx.org/en/latest/developers/how-tos/get-ready-for-python-dev.html>_ in this repo.
All tools in edx_django_utils should expose their public api in their init.py files. This entails adding to init.py all functions/classes/constants/objects that are intended to be used by users of library.
If you're having trouble, we have discussion forums at
discuss.openedx.org <https://discuss.openedx.org>_ where you can connect with others in the
community.
Our real-time conversations are on Slack. You can request a Slack invitation, then join our community Slack workspace.
For anything non-trivial, the best path is to open an issue__ in this
repository with as many details about the issue you are facing as you
can provide.
__ https://github.com/openedx/django-config-models/issues
For more information about these options, see the Getting Help_ page.
.. _Slack invitation: https://openedx.org/slack .. _community Slack workspace: https://openedx.slack.com/ .. _Getting Help: https://openedx.org/getting-help
Contributions are very welcome.
Please read How To Contribute <https://github.com/openedx/.github/blob/master/CONTRIBUTING.md>_ for details.
PR description template should be automatically applied if you are sending PR from github interface; otherwise you
can find it it at PULL_REQUEST_TEMPLATE.md <https://github.com/openedx/edx-django-utils/blob/master/.github/PULL_REQUEST_TEMPLATE.md>_
Issue report template should be automatically applied if you are sending it from github UI as well; otherwise you
can find it at ISSUE_TEMPLATE.md <https://github.com/openedx/edx-django-utils/blob/master/.github/ISSUE_TEMPLATE.md>_
This project is currently accepting all types of contributions, bug fixes, security fixes, maintenance work, or new features. However, please make sure to have a discussion about your new feature idea with the maintainers prior to beginning development to maximize the chances of your change being accepted. You can start a conversation by creating a new issue on this repo summarizing your idea.
All community members are expected to follow the Open edX Code of Conduct_.
.. _Open edX Code of Conduct: https://openedx.org/code-of-conduct/
The assigned maintainers for this component and other project details may be
found in Backstage_. Backstage pulls this data from the catalog-info.yaml
file in this repo.
.. _Backstage: https://backstage.openedx.org/catalog/default/component/edx-django-utils
Please do not report security issues in public. Please email security@openedx.org.
The code in this repository is licensed under the Apache License, Version 2.0, unless otherwise noted.
Please see LICENSE.txt for details.
.. |pypi-badge| image:: https://img.shields.io/pypi/v/edx-django-utils.svg :target: https://pypi.python.org/pypi/edx-django-utils/ :alt: PyPI
.. |ci-badge| image:: https://github.com/openedx/edx-django-utils/workflows/Python%20CI/badge.svg?branch=master :target: https://github.com/openedx/edx-django-utils/actions?query=workflow%3A%22Python+CI%22 :alt: CI
.. |codecov-badge| image:: http://codecov.io/github/edx/edx-django-utils/coverage.svg?branch=master :target: http://codecov.io/github/edx/edx-django-utils?branch=master :alt: Codecov
.. |doc-badge| image:: https://readthedocs.org/projects/edx-django-utils/badge/?version=latest :target: http://edx-django-utils.readthedocs.io/en/latest/ :alt: Documentation
.. |pyversions-badge| image:: https://img.shields.io/pypi/pyversions/edx-django-utils.svg :target: https://pypi.python.org/pypi/edx-django-utils/ :alt: Supported Python versions
.. |license-badge| image:: https://img.shields.io/github/license/edx/edx-django-utils.svg :target: https://github.com/openedx/edx-django-utils/blob/master/LICENSE.txt :alt: License
.. |status-badge| image:: https://img.shields.io/badge/Status-Maintained-brightgreen :alt: Maintenance status
.. All enhancements and patches to edx_django_utils will be documented in this file. It adheres to the structure of https://keepachangelog.com/ , but in reStructuredText instead of Markdown (for ease of incorporation into Sphinx documentation and the PyPI description).
This project adheres to Semantic Versioning (https://semver.org/).
.. There should always be an "Unreleased" section for changes pending release.
Removed
* Removed deprecated monitoring middleware Classes
7.1.0 - 2024-12-05
------------------
Added
~~~~~
* Added signals monitoring_support_process_request, monitoring_support_process_response, and monitoring_support_process_exception to the MonitoringSupportMiddleware to enable plugging in monitoring capabilities.
7.0.1 - 2024-11-21
------------------
Fixed
~~~~~
* Fix bug where code owner custom attributes were being defined, even when the CODE_OWNER_MAPPINGS setting was not defined.
7.0.0 - 2024-10-16
------------------
Removed
background_task monitoring function.get_current_transaction (used internally only) from the public API.Changed
* Added Datadog implementation of ``set_monitoring_transaction_name`` and refactored the functionality.
[6.0.0] - 2024-10-09
---------------------
Added
~~~~~
* Added support for python3.12
* Dropped support for python<3.12 versions
[5.16.0] - 2024-09-27
---------------------
Added
~~~~~
* Added a new method to backends for ``tag_root_span_with_error`` and added Datadog implementation of the functionality.
* Uses the new method to tag the root span when processing exceptions.
Changed
CachedCustomMonitoringMiddleware to MonitoringSupportMiddleware and deprecated the old name. It will be removed in a future release.Added
* Added Datadog implementation of ``function_trace`` and allowed implementation to be configurable.
[5.14.2] - 2024-05-31
---------------------
Fixed
edx_django_utils.monitoring.enable_frontend_monitoring_middleware is enabled.Fixed
* Added default value while getting content-type header to avoid KeyError.
[5.14.0] - 2024-05-22
---------------------
Added
FrontendMonitoringMiddleware for inserting frontend monitoring HTML script tags to response, configured by new Django setting OPENEDX_TELEMETRY_FRONTEND_SCRIPTS.Added
* Initial support for sending monitoring data to OpenTelemetry collector or Datadog agent, configured by new Django setting ``OPENEDX_TELEMETRY``. See monitoring README for details.
[5.12.0] - 2024-03-29
---------------------
Added
Python 3.11Added
* Added support for ``Python 3.12``
Removed
Django 3.2Added
* Added manufacture_data management command
[5.9.0] - 2023-11-27
--------------------
Removed
edx_django_utils.cache.disable_forced_cache_miss_for_none which was added in 5.7.0.Changed
* Adjusted ``get_plugin_apps`` to log at info level rather than debug and with more detail, though with a comment that this may not actually end up logging.
[5.7.0] - 2023-08-04
--------------------
Added
~~~~~
* Support added for Django 4.2
Fixed
~~~~~
* Fixed bug where None was not properly being stored by TieredCache.
For backward compatibility, ``edx_django_utils.cache.disable_forced_cache_miss_for_none`` waffle switch has
been added, which defaults to the old broken behavior of treating None as a cache miss.
[5.6.0] - 2023-07-24
--------------------
Changed
Changed
* Switched to ``sphinx-book-theme`` as the new standard theme across all Open
edX repos. See https://github.com/openedx/edx-sphinx-theme/issues/184 for
more details.
* CookieMonitoringMiddleware will now remove cookies based on a ``COOKIE_PREFIXES_TO_REMOVE`` setting
[5.4.0] - 2023-04-12
--------------------
Added
~~~~~
* Added Content-Security-Policy response header middleware under ``security/csp``
[5.3.0] - 2023-03-17
--------------------
Fixed
~~~~~
* Report both specified and existing email when refusing to create a user
[5.2.0] - 2022-10-06
--------------------
Added
~~~~~
* Added a wrapper for background_task in monitoring.
[5.1.0] - 2022-09-19
--------------------
Added
~~~~~
* Utilities for safely determining the IP address of a request: ``edx_django_utils.ip`` (moved from edx-platform)
[5.0.1] - 2022-09-09
--------------------
Changed
Changed
* Corrupt cookie logging:
* Make independent of other cookie logging; no longer needs to meet cookie size threshold or sampling rate.
* **Breaking change**, although low impact: Setting name changed from ``UNUSUAL_COOKIE_SAMPLING_PUBLIC_KEY`` to ``UNUSUAL_COOKIE_HEADER_PUBLIC_KEY``.
* New setting ``UNUSUAL_COOKIE_HEADER_LOG_CHUNK`` helps avoid truncated (non-decryptable) messages by splitting the output across multiple log messages.
[4.8.1] - 2022-05-06
--------------------
Added
~~~~~
* Added ability to log headers when a corrupted cookie is detected in a large (or sampled) cookie header
[4.8.0] - 2022-05-06
--------------------
Bad version -- tag does not match package version, not released to PyPI. Released as 4.8.1 instead.
[4.7.0] - 2022-05-05
--------------------
Added
~~~~~
* Added ``encrypt_for_log`` logging helper and ``log-sensitive`` CLI command for encrypted logging of sensitive information
[4.6.0] - 2022-03-16
--------------------
Added
~~~~~
* Added ``CookieMonitoringMiddleware`` for monitoring cookie header sizes and cookie sizes.
[4.5.0] - 2022-01-31
--------------------
Removed
Fixed
* No longer clear the ``RequestCache`` during the exception-handling phase (wait until response phase)
* It turns out all the ``process_exception`` methods get called until one returns a response, and only *then* do the ``process_response`` methods start getting called. The result was that on exception, some middlewares were unable to use RequestCache'd values in their response phase.
Updated
Updated
* Replaced usage of 'django.conf.urls.url()' with 'django.urls.re_path()'
[4.4.0] - 2021-09-02
--------------------
Added
~~~~~
* Added ``DeploymentMonitoringMiddleware`` to record ``Python`` and ``Django`` versions in NewRelic with each transaction.
[4.3.0] - 2021-07-20
--------------------
Added
~~~~~
* Added user and group management utilities.
[4.2.0] - 2021-07-14
--------------------
Added
~~~~~
* Added support for Django 3.1 and 3.2
[4.1.0] - 2021-06-01
--------------------
Added
~~~~~
* Added mixin for a custom Django admin class which disables CRUD operation on the admin's model.
Added
~~~~~
* Script new_relic_nrql_search.py to search the NRQL in New Relic alert policies and dashboards using a supplied regex.
[4.0.0] - 2021-05-03
--------------------
Removed
CodeOwnerMonitoringMiddleware. It had moved in a past commit. Although technically a breaking change, all references in the Open edX platform have already been updated to point to the new location.Added
* Added new ``code_owner_theme`` and ``code_owner_squad`` custom attributes. This is useful in cases where the ``code_owner`` combines a theme and squad name, because monitoring can instead reference ``code_owner_squad`` to be resilient to theme name updates. For the decision doc, see edx_django_utils/monitoring/docs/decisions/0004-code-owner-theme-and-squad.rst.
Updated
Added
* Added ``pluggable_override`` decorator.
[3.15.0] - 2021-03-02
---------------------
* Added chunked_queryset utility.
[3.14.0] - 2020-12-15
---------------------
Removed
Added
* Added record_exception to monitor caught exceptions.
Updated
deprecated_monitoring_utils custom attribute values to make it simpler to track down usage.Added
* Added set_code_owner_attribute decorator for use with celery tasks.
* Added set_code_owner_attribute_from_module as an alternative to the decorator.
Updated
Added
* Added ADR 0004-public-api-and-app-organization.rst to explain a new app organization, which makes use of the public API more consistent.
Updated
Deprecated
* Deprecated the old locations of CachedCustomMonitoringMiddleware, CodeOwnerMonitoringMiddleware, and MonitoringMemoryMiddleware.
* Deprecated various methods from modules that were always meant to be used from the public API.
* accumulate
* increment
* set_custom_attribute
* set_custom_attributes_for_course_key
* Added additional custom attributes for deprecated classes and methods to make them safer to retire.
.. note::
Some method implementations that were available in the public API were moved without adding a deprecated equivalent. These were not found when searching, so hopefully they are only used via the public API, which did not change. This includes functions in ``transactions.py`` and ``code_owner/utils.py``.
Removed
~~~~~~~
* Removed the middleware ordering checks. This is not a typical Django feature and it is painful when refactoring.
[3.10.0] - 2020-10-28
---------------------
Added
~~~~~
* Added logging filter classes for users and remote IP addresses to be used by all IDAs. These were moved here from edx-platform.
[3.9.0] - 2020-10-21
--------------------
Updated
~~~~~~~
* Exposed existing get_code_owner_from_module via the public api.
* Fixed get_code_owner_from_module to not require a call to is_code_owner_mappings_configured beforehand.
* Set the existing code_owner_path_module custom attribute, even for cases where the transaction name was used, rather than the view module.
* Refactor code owner setting processing.
[3.8.0] - 2020-08-31
--------------------
Updated
~~~~~~~
* Renamed "custom metric" to "custom attribute" throughout the monitoring library. This decision can be read about in the ADR 0002-custom-monitoring-language.rst. The following have been deprecated:
* set_custom_metric (use set_custom_attribute)
* set_custom_metrics_for_course_key (use set_custom_attributes_for_course_key)
* MonitoringCustomMetricsMiddleware (use CachedCustomMonitoringMiddleware)
* CachedCustomMonitoringMiddleware.accumulate_metric (use CachedCustomMonitoringMiddleware.accumulate_attribute)
* This wasn't meant to be used publicly, but was deprecated just in case.
* CodeOwnerMetricMiddleware (use CodeOwnerMonitoringMiddleware)
[3.7.4] - 2020-08-29
--------------------
* Fix to custom monitoring accumulate to actually accumulate rather than overwrite.
[3.7.3] - 2020-08-12
--------------------
Updated
~~~~~~~
* Upgrade psutil to latest version
[3.7.2] - 2020-08-10
--------------------
Updated
~~~~~~~
* Added missing classes to plugins public api. See ``plugins.__init__.py`` for latest api.
* Updated plugin method names to be more descriptive. See ``plugins.__init__.py`` for latest.
.. note:: Although these changes are backwards incompatible, they are being added as a bug fix because plugins code release (3.7.0) is not yet in use.
[3.7.1] - 2020-08-10
--------------------
Updated
~~~~~~~
* Exposing all public functions in edx_django_utils/plugins directory in its __init__.py file.
* this was done to keep inline with standard/pattern used in other packages in edx_django_utils
[3.7.0] - 2020-08-10
--------------------
Added
~~~~~
* Adding Plugin infrastructure
* Allows IDAs to use plugins
[3.6.0] - 2020-08-04
--------------------
Added
~~~~~
* Improved documentation for CodeOwnerMetricMiddleware, including a how_tos/add_code_owner_custom_metric_to_an_ida.rst for adding it to a new IDA.
* Added ignore_transaction monitoring utility to ignore transactions we don't want tracked.
Updated
~~~~~~~
* Moved transaction-related monitoring code into it's own file. Still exposed through `__init__.py` so it's a non-breaking change.
[3.5.0] - 2020-07-22
--------------------
Updated
~~~~~~~
* Added a catch-all capability to CodeOwnerMetricMiddleware when CODE_OWNER_MAPPINGS includes a '*' as a team's module. The catch-all is used only if there is no other match.
[3.4.0] - 2020-07-20
--------------------
Added
~~~~~
* Added get_current_transaction for monitoring that returns a transaction object with a name property.
Updated
~~~~~~~
* Updated CodeOwnerMetricMiddleware to use NewRelic's current transaction for cases where resolve() doesn't work to determine the code_owner, like for Middleware.
[3.3.0] - 2020-07-16
--------------------
Added
~~~~~
* CodeOwnerMetricMiddleware was moved here (from edx-platform) in order to be able to take advantage of the ``code_owner`` metric in other IDAs. For details on this decision, see the `ADR for monitoring code owner`_. See the docstring for more details on usage.
.. _ADR for monitoring code owner: https://github.com/openedx/edx-django-utils/blob/master/edx_django_utils/monitoring/docs/decisions/0001-monitoring-by-code-owner.rst
[3.2.3] - 2020-05-30
--------------------
* Removed ceninusepy3 usage.
[3.2.2] - 2020-05-04
--------------------
* Added support for python 3.8 and dropped support for Django versions older than 2.2
[3.2.1] - 2020-04-17
--------------------
Changed
~~~~~~~
* imported get_cache_key in cache/__init__.py.
[3.2.0] - 2020-04-09
--------------------
Added
~~~~~
* Added get_cache_key utility.
[2.0.1] - 2019-10-09
--------------------
Changed
~~~~~~~
* Fixed: Updated function tracing to accomodate changes in New Relic's 5.x Agent.
[2.0.0] - 2019-07-07
--------------------
Changed
~~~~~~~
* Converted Middleware (from old style MIDDLEWARE_CLASSES to MIDDLEWARE).
* Removed support for Django versions < 1.11
[1.0.1] - 2018-09-07
--------------------
Changed
~~~~~~~
* Fixed: RequestCache now properly uses thread.local.
* Fixed: CachedResponse.__repr__ now handles unicode.
[1.0.0] - 2018-08-28
--------------------
Added
~~~~~~~
* Add ``data`` dict property to better match legacy RequestCache interface.
Changed
~~~~~~~
* Change is_hit/is_miss to is_found.
[0.5.1] - 2018-08-17
--------------------
Changed
~~~~~~~
* Fixed bug in TieredCacheMiddleware dependency declaration.
[0.5.0] - 2018-08-16
--------------------
Changed
~~~~~~~
* Restored Python 3 support.
* Refactor/clean-up, including Middleware dependency checking.
* Docs updates and other cookiecutter updates.
[0.4.1] - 2018-08-10
--------------------
Changed
~~~~~~~
* Split out TieredCacheMiddleware from RequestCacheMiddleware.
[0.4.0] - 2018-08-10
--------------------
Changed
~~~~~~~
* Rename CacheUtilsMiddleware to RequestCacheMiddleware.
[0.3.0] - 2018-08-02
--------------------
Removed
~~~~~~~
* Temporarily dropped Python 3 support to land this.
[0.2.0] - 2018-08-01
--------------------
Added
~~~~~
* Added cache and monitoring utilities.
[0.1.0] - 2018-07-23
--------------------
Added
~~~~~
* First release on PyPI.
FAQs
EdX utilities for Django Application development.
We found that edx-django-utils demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Research
Security News
Malicious Go packages are impersonating popular libraries to install hidden loader malware on Linux and macOS, targeting developers with obfuscated payloads.
Security News
Bybit's $1.46B hack by North Korea's Lazarus Group pushes 2025 crypto losses to $1.6B in just two months, already surpassing all of 2024's $1.49B total.
Security News
OpenSSF has published OSPS Baseline, an initiative designed to establish a minimum set of security-related best practices for open source software projects.