aiodiskqueue

============ aiodiskqueue

Persistent queue for Python AsyncIO.

|release| |python| |tests| |codecov| |docs| |pre-commit| |Code style: black|

Description

This library provides a persistent FIFO queue for Python AsyncIO:

• Queue content persist a process restart
• Feature parity with Python's asyncio.Queue <https://docs.python.org/3/library/asyncio-queue.html#queue>_
• Similar API to Python's asyncio.Queue <https://docs.python.org/3/library/asyncio-queue.html#queue>_
• Sane logging
• Type hints
• Fully tested
• Supports different storage engines and can be extended with custom storage engines

Usage

Here is a basic example on how to use the queue:

.. code:: python

import asyncio
from aiodiskqueue import Queue

async def main():
    q = await Queue.create("example_queue.sqlite")
    await q.put("some item")
    item = await q.get()
    print(item)

asyncio.run(main())

Please see the examples folder for more usage examples.

Installation

You can install this library directly from PyPI with the following command:

.. code:: shell

pip install aiodiskqueue

Logging

The name of the logger for all logging by this library is: aiodiskqueue.

Storage Engines

aiodiskqueue support different storage engines. The default engine is DbmEngine.

We measured the throughput for a typical load scenario (5 producers, 1 consumer) with each storage engine:

.. image:: https://imgpile.com/images/9luzXk.png :width: 800 :alt: Measurements

• DbmEngine: Consistent throughput at low and high volumes and about 3 x faster then Sqlite
• PickledList: Very fast at low volumes, but does not scale well
• SqliteEngine: Consistent throughput at low and high volumes. Relatively slow.

The scripts for running the measurements and generating this chart can be found in the measurements folder.

.. |release| image:: https://img.shields.io/pypi/v/aiodiskqueue?label=release :target: https://pypi.org/project/aiodiskqueue/

.. |python| image:: https://img.shields.io/pypi/pyversions/aiodiskqueue :target: https://pypi.org/project/aiodiskqueue/

.. |tests| image:: https://github.com/ErikKalkoken/aiodiskqueue/actions/workflows/main.yml/badge.svg :target: https://github.com/ErikKalkoken/aiodiskqueue/actions

.. |codecov| image:: https://codecov.io/gh/ErikKalkoken/aiodiskqueue/branch/main/graph/badge.svg?token=V43h7hl1Te :target: https://codecov.io/gh/ErikKalkoken/aiodiskqueue

.. |docs| image:: https://readthedocs.org/projects/aiodiskqueue/badge/?version=latest :target: https://aiodiskqueue.readthedocs.io/en/latest/?badge=latest

.. |pre-commit| image:: https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit&logoColor=white :target: https://github.com/pre-commit/pre-commit

.. |Code style: black| image:: https://img.shields.io/badge/code%20style-black-000000.svg :target: https://github.com/psf/black