lox

.. image:: https://raw.githubusercontent.com/BrianPugh/lox/main/assets/lox_200w.png

.. image:: https://img.shields.io/pypi/v/lox.svg :target: https://pypi.python.org/pypi/lox

.. image:: https://circleci.com/gh/BrianPugh/lox.svg?style=svg :target: https://circleci.com/gh/BrianPugh/lox

.. image:: https://readthedocs.org/projects/lox/badge/?version=latest :target: https://lox.readthedocs.io/en/latest/?badge=latest :alt: Documentation Status

Threading and multiprocessing made easy.

• Free software: Apache-2.0 license
• Documentation: https://lox.readthedocs.io.
• Python >=3.6

Lox provides decorators and synchronization primitives to quickly add concurrency to your projects.

Installation

pip3 install --user lox

Features

• Multithreading: Powerful, intuitive multithreading in just 2 additional lines of code.

• Multiprocessing: Truly parallel function execution with the same interface as multithreading.

• Synchronization: Advanced thread synchronization, communication, and resource management tools.

Todos

• All objects except lox.process are for threads. These will eventually be multiprocess friendly.

Usage

Easy Multithreading ^^^^^^^^^^^^^^^^^^^

>>> import lox
>>>
>>> @lox.thread(4) # Will operate with a maximum of 4 threads
... def foo(x,y):
...     return x*y
>>> foo(3,4) # normal function calls still work
12
>>> for i in range(5):
...     foo.scatter(i, i+1)
-ignore-
>>> # foo is currently being executed in 4 threads
>>> results = foo.gather() # block until results are ready
>>> print(results) # Results are in the same order as scatter() calls
[0, 2, 6, 12, 20]

Or, for example, if you aren't allowed to directly decorate the function you would like multithreaded/multiprocessed, you can just directly invoke the decorator:

.. code-block:: pycon

>>> # Lets say we don't have direct access to this function
... def foo(x, y):
...     return x * y
...
>>>
>>> def my_func():
...     foo_threaded = lox.thread(foo)
...     for i in range(5):
...         foo_threaded.scatter(i, i + 1)
...     results = foo_threaded.gather()
...     # foo is currently being executed in default 50 thread executor pool
...     return results
...

This also makes it easier to dynamically control the number of thread/processes in the executor pool. The syntax is a little weird, but this is just explicitly invoking a decorator that has optional arguments:

.. code-block:: pycon

>>> # Set the number of executer threads to 10
>>> foo_threaded = lox.thread(10)(foo)

Easy Multiprocessing ^^^^^^^^^^^^^^^^^^^^

.. code-block:: pycon

>>> import lox
>>>
>>> @lox.process(4)  # Will operate with a pool of 4 processes
... def foo(x, y):
...     return x * y
...
>>> foo(3, 4)  # normal function calls still work
12
>>> for i in range(5):
...     foo.scatter(i, i + 1)
...
-ignore-
>>> # foo is currently being executed in 4 processes
>>> results = foo.gather()  # block until results are ready
>>> print(results)  # Results are in the same order as scatter() calls
[0, 2, 6, 12, 20]

Progress Bar Support (tqdm) ^^^^^^^^^^^^^^^^^^^^^^^^^^^

.. code-block:: pycon

>>> import lox
>>> from random import random
>>> from time import sleep
>>>
>>> @lox.thread(2)
... def foo(multiplier):
...     sleep(multiplier * random())
...
>>> for i in range(10):
>>>     foo.scatter(i)
>>> results = foo.gather(tqdm=True)
90%|████████████████████████████████▌        | 9/10 [00:03<00:00,  1.32it/s]
100%|███████████████████████████████████████| 10/10 [00:06<00:00,  1.46s/it]

======= History

0.11.0 (2022-04-07)

• Set number of workers to 0 (in thread execution) if the environment variable LOX_DEBUG is set to a true-like value (true, 1, etc.). Makes it easier to set breakpoints in multithreaded code without having to manually edit the decorator.

0.10.0 (2021-12-18)

• Remove dependency pinning.
• Allow @lox.thread(0). This will execute scatter calls in parent thread. Useful for debugging breakpoints in parallelized code.

0.9.0 (2020-11-25)

• tqdm support on lox.process.gather. See v0.8.0 release notes for usage.

0.8.0 (2020-11-25)

• tqdm support on lox.thread.gather

  • Can be a bool::

    >>> my_func.gather(tqdm=True)
    

  • Can be a tqdm object::

    >>> from tqdm import tqdm
    >>> pbar = tqdm(total=100)
    >>> for _ in range(100):
    >>>     my_func.scatter()
    >>> my_func.gather(tqdm=pbar)
    

0.7.0 (2020-07-20)

• Complete rework of workers
  • Fix memory leaks
• Drop support for python3.5
• Drop support for chaining in favor of simpler codebase

0.6.3 (2019-07-30)

• Alternative fix for 0.6.2.

0.6.2 (2019-07-21)

• Update dependencies
• Fix garbage-collecting exclusiviity

0.6.1 (2019-07-21)

• Fix memory leak in lox.process.

0.6.0 (2019-07-21)

• lox.Announcement subscribe() calls now return another Announcement object that behaves like a queue instead of an actual queue. Allows for many-queue-to-many-queue communications.

• New Object: lox.Funnel. allows for waiting on many queues for a complete set of inputs indicated by a job ID.

0.5.0 (2019-07-01)

• New Object: lox.Announcement. Allows a one-to-many thread queue with backlog support so that late subscribers can still get all (or most recent) announcements before they subscribed.

• New Feature: lox.thread scatter calls can now be chained together. scatter now returns an int subclass that contains metadata to allow chaining. Each scatter call can have a maximum of 1 previous scatter result.

• Documentation updates, theming, and logos

0.4.3 (2019-06-24)

• Garbage collect cached decorated object methods

0.4.2 (2019-06-23)

• Fixed multiple instances and successive scatter and gather calls to wrapped methods

0.4.1 (2019-06-23)

• Fixed broken workers and unit tests for workers

0.4.0 (2019-06-22)

• Semi-breaking change: lox.thread and lox.process now automatically pass the object instance when decorating a method.

0.3.4 (2019-06-20)

• Print traceback in red when a thread crashes

0.3.3 (2019-06-19)

• Fix bug where thread in scatter of lox.thread double releases on empty queue

0.3.2 (2019-06-17)

• Fix manifest for installation from wheel

0.3.1 (2019-06-17)

• Fix package on pypi

0.3.0 (2019-06-01)

• Multiprocessing decorator. lox.pool renamed to lox.thread

• Substantial pytest bug fixes

• Documentation examples

• timeout for RWLock

0.2.1 (2019-05-25)

• Fix IndexSemaphore context manager

0.2.0 (2019-05-24)

• Added QLock

• Documentation syntax fixes

0.1.1 (2019-05-24)

• CICD test

0.1.0 (2019-05-24)

• First release on PyPI.