edX Event Bus - Kafka #####################
Kafka implementation for Open edX event bus.
|pypi-badge| |ci-badge| |codecov-badge| |doc-badge| |pyversions-badge| |license-badge|
Overview
This package implements an event bus for Open edX using Kafka.
The event bus acts as a broker between services publishing events and other services that consume these events. Implementing the event bus will allow for asynchronous messaging across services which greatly improves efficiency as we don't have to wait for scheduled batch synchronizations. Additionally, since the services all speak to the event bus, they are independent of one another and can still function if one service crashes.
This package contains both the publishing code, which processes events into messages to send to the broker, and the consumer code,
which polls the broker using a while True loop in order to turn messages back into event data to be emitted.
The actual Kafka host will be configurable.
The goal for event-bus-kafka is to eventually have a flexible event bus that can be easily brought into other apps and repositories to produce and consume arbitrary topics. Ideally, the event bus itself will also be an abstraction behind which platform maintainers can use non-Kafka implementations (Redis, Pulsar, etc.). The documentation/ADRs may also be moved to more appropriate places as the process matures.
The repository works together with the openedx/openedx-events repository to make the fully functional event bus.
Documentation
To use this implementation of the Event Bus with openedx-events, you'll need to ensure that you include the dependency confluent_kafka[avro,schema-registry] (see ADR 5 <https://github.com/openedx/event-bus-kafka/blob/main/docs/decisions/0005-optional-import-of-confluent-kafka.rst>_ for an explanation) and set the following Django settings::
EVENT_BUS_KAFKA_BOOTSTRAP_SERVERS: ...
EVENT_BUS_KAFKA_SCHEMA_REGISTRY_URL: ...
EVENT_BUS_TOPIC_PREFIX: ...
# On the producing side:
EVENT_BUS_PRODUCER: edx_event_bus_kafka.create_producer
# On the consuming side:
EVENT_BUS_CONSUMER: edx_event_bus_kafka.KafkaEventConsumer
Optional settings that are worth considering:
EVENT_BUS_KAFKA_CONSUMER_CONSECUTIVE_ERRORS_LIMITFor manual testing, see <docs/how_tos/manual_testing.rst>__.
Django management commands:
produce_event management commandconsume_events management commandOEP-52 documentation: https://open-edx-proposals.readthedocs.io/en/latest/architectural-decisions/oep-0052-arch-event-bus-architecture.html
(TODO: Set up documentation <https://openedx.atlassian.net/wiki/spaces/DOC/pages/21627535/Publish+Documentation+on+Read+the+Docs>_)
Development Workflow
.. code-block::
git clone git@github.com:openedx/event-bus-kafka.git cd event-bus-kafka
mkvirtualenv -p python3.8 event-bus-kafka
.. code-block::
workon event-bus-kafka
git checkout main git pull
make requirements
make validate
git checkout -b <your_github_username>/<short_description>
vim …
pytest ./path/to/new/tests
make validate
git commit … git push
Installation
This library is not intended to be a direct dependency of any service. Instead, it will likely be installed by some private mechanism. Unless the platform gains new processes for installing private dependencies, upgrades will need to be manually managed via a pin.
License
The code in this repository is licensed under the AGPL 3.0 unless otherwise noted.
Please see LICENSE.txt <LICENSE.txt>_ for details.
How To Contribute
Contributions are very welcome.
Please read How To Contribute <https://github.com/openedx/.github/blob/master/CONTRIBUTING.md>_ for details.
The pull request description template should be automatically applied if you are creating a pull request from GitHub. Otherwise you
can find it at PULL_REQUEST_TEMPLATE.md <.github/PULL_REQUEST_TEMPLATE.md>_.
The issue report template should be automatically applied if you are creating an issue on GitHub as well. Otherwise you
can find it at ISSUE_TEMPLATE.md <.github/ISSUE_TEMPLATE.md>_.
Reporting Security Issues
Please do not report security issues in public. Please email security@openedx.org.
Getting Help
If you're having trouble, we have discussion forums at https://discuss.openedx.org where you can connect with others in the community.
Our real-time conversations are on Slack. You can request a Slack invitation, then join our community Slack workspace.
For more information about these options, see the Getting Help_ page.
.. _Slack invitation: https://openedx-slack-invite.herokuapp.com/ .. _community Slack workspace: https://openedx.slack.com/ .. _Getting Help: https://openedx.org/getting-help
.. |pypi-badge| image:: https://img.shields.io/pypi/v/edx-event-bus-kafka.svg :target: https://pypi.python.org/pypi/edx-event-bus-kafka/ :alt: PyPI
.. |ci-badge| image:: https://github.com/openedx/event-bus-kafka/workflows/Python%20CI/badge.svg?branch=main :target: https://github.com/openedx/event-bus-kafka/actions :alt: CI
.. |codecov-badge| image:: https://codecov.io/github/openedx/event-bus-kafka/coverage.svg?branch=main :target: https://codecov.io/github/openedx/event-bus-kafka?branch=main :alt: Codecov
.. |doc-badge| image:: https://readthedocs.org/projects/edx-event-bus-kafka/badge/?version=latest :target: https://edx-event-bus-kafka.readthedocs.io/en/latest/ :alt: Documentation
.. |pyversions-badge| image:: https://img.shields.io/pypi/pyversions/edx-event-bus-kafka.svg :target: https://pypi.python.org/pypi/edx-event-bus-kafka/ :alt: Supported Python versions
.. |license-badge| image:: https://img.shields.io/github/license/openedx/event-bus-kafka.svg :target: https://github.com/openedx/event-bus-kafka/blob/main/LICENSE.txt :alt: License
Change Log ##########
.. All enhancements and patches to edx_event_bus_kafka will be documented in this file. It adheres to the structure of https://keepachangelog.com/ , but in reStructuredText instead of Markdown (for ease of incorporation into Sphinx documentation and the PyPI description).
This project adheres to Semantic Versioning (https://semver.org/).
.. There should always be an "Unreleased" section for changes pending release.
Unreleased
[6.0.0] - 2024-08-24
[5.8.1] - 2024-08-02
kafka_received_message to track whether a message was processed or not.[5.8.0] - 2024-08-01
consumer.consume.[5.7.0] - 2024-03-22
[5.6.0] - 2024-01-25
[5.5.0] - 2023-09-21
[5.4.0] - 2023-08-28
[5.3.1] - 2023-08-10
[5.3.0] - 2023-08-08
[5.2.0] - 2023-08-03
[5.1.0] - 2023-05-17
[5.0.0] - 2023-05-17
signal argument from consumer initialization[4.0.1] - 2023-05-10
signal back as an argument when creating a consumer for compatibility with the openedx-events API[4.0.0] - 2023-05-10
consume_events management command as this command will be provided by openedx_events. To replay events using the
openedx-events version of the management command, pass --extra '{"offset_time": "2023-01-08T06:46:22"}' instead of -o 2023-01-08T06:46:22.offset_timestamp argument has been removed from consume_indefinitely and reset_offsets_and_sleep_indefinitely methods.
It is now added as an optional argument named offset_time to KafkaEventConsumer constructor.[3.10.0] - 2023-05-05
edx-sphinx-theme to sphinx-book-theme since the former is
deprecated[3.9.6] - 2023-02-24
[3.9.5] - 2023-02-24
consume_events command now loads all public signals so that the consumer can load signals by event_type even if their modules were not already imported[3.9.4] - 2023-02-16
[3.9.3] - 2023-02-10
message_id in audit log when message is produced (was None)[3.9.2] - 2023-02-08
[3.9.1] - 2023-02-07
There was no version 3.9.0, due to a release issue. (Ignore any v3.9.0 tag.)
[3.8.1] - 2023-02-03
[3.8.0] - 2023-01-31
[3.7.1] - 2023-01-31
[3.7.0] - 2023-01-30
reset_offsets_and_sleep_indefinitely method to consumer; relying code should switch to calling this when an offset timestamp is set.offset_timestamp parameter on the consumer's consume_indefinitely method, since reset_offsets_and_sleep_indefinitely should be used instead.[3.6.3] - 2023-01-27
[3.6.2] - 2023-01-26
[3.6.1] - 2023-01-20
[3.6.0] - 2023-01-06
[3.5.1] - 2023-01-06
[3.5.0] - 2023-01-05
EVENT_BUS_KAFKA_CONSUMER_CONSECUTIVE_ERRORS_LIMIT will terminate the consumer if too many errors occur in a row, supporting an automated version of "have you tried turning it off and on again" (as long as consumer will automatically be restarted e.g. by Kubernetes).[3.4.1] - 2022-12-20
[3.4.0] - 2022-12-16
[3.3.0] - 2022-12-15
[3.2.0] - 2022-12-14
[3.1.0] - 2022-12-07
[3.0.0] - 2022-12-02
[2.1.0] - 2022-12-01
[2.0.0] - 2022-11-28
get_producer and rename to create_producer, as we now rely on the wrapper in openedx-events to cache that callUpgrading library from 1.x:
edx_event_bus_kafka.get_producer with openedx_events.event_bus.get_producerEVENT_BUS_PRODUCER = "edx_event_bus_kafka.create_producer"These breaking changes are only relevant for the producing side. (This should only include the CMS at the moment.)
[1.10.0] - 2022-11-21
[1.9.0] - 2022-11-15
[1.8.1] - 2022-11-10
[1.8.0] - 2022-11-09
[1.7.0] - 2022-11-04
[1.6.0] - 2022-11-04
[1.5.0] - 2022-11-01
[1.4.3] - 2022-10-31
[1.4.2] - 2022-10-31
[1.4.1] - 2022-10-28
[1.4.0] - 2022-10-21
[1.3.0] - 2022-10-20
effort field)[1.2.0] - 2022-10-13
EVENT_BUS_KAFKA_CONSUMERS_ENABLED now defaults to True instead of False[1.1.0] - 2022-10-06
[1.0.0] - 2022-10-03
[0.7.0] - 2022-09-08
EventProducerKafka is now KafkaEventProducer[0.6.2] - 2022-09-08
EVENT_BUS_TOPIC_PREFIX[0.6.1] - 2022-09-06
EVENT_BUS_KAFKA_POLL_INTERVAL_SEC.[0.6.0] - 2022-09-01
edx_event_bus_kafka package and edx_event_bus_kafka.management.commands package; all other modules should be considered unstable and not for external use.[0.5.1] - 2022-08-31
__init__.py files.)[0.5.0] - 2022-08-31
Breaking changes in the producer module, refactored to expose a better API:
send_to_event_bus(...), relying code should now call get_producer().send(...).sync kwarg is gone; to flush and sync messages before shutdown, call get_producer().prepare_for_shutdown() instead.Clarify that config module is for internal use only.
Implementation changes: Only a single Producer is created, and is used for all signals.
[0.4.4] - 2022-08-26
[0.4.3] - 2022-08-24
<https://github.com/openedx/event-bus-kafka/issues/16>__.[0.4.2] - 2022-08-24
[0.4.1] - 2022-08-18
Remove confluent-kafka as a formal dependency of the repository.
Add an ADR to explain why this work was done.
[0.4.0] - 2022-08-15
Rename settings to have consistent prefix.
KAFKA_CONSUMERS_ENABLED becomes EVENT_BUS_KAFKA_CONSUMERS_ENABLEDCONSUMER_POLL_TIMEOUT becomes EVENT_BUS_KAFKA_CONSUMER_POLL_TIMEOUT[0.3.1] - 2022-08-11
[0.3.0] - 2022-08-10
[0.2.0] - 2022-08-09
[0.1.0] - 2022-06-16
FAQs
Kafka implementation for Open edX event bus.
We found that edx-event-bus-kafka demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
PyPI confirms no security flaws were exploited in the Ultralytics supply chain attack and highlights improvements for safer package publishing.
Security News
Research
The Socket Research Team breaks down a malicious wrapper package that uses obfuscation to harvest credentials and exfiltrate sensitive data.
Research
Security News
Attackers used a malicious npm package typosquatting a popular ESLint plugin to steal sensitive data, execute commands, and exploit developer systems.