Django based tool for monitoring huey task queue: https://github.com/coleifer/huey
Django based tool for monitoring huey task queue
Current implementation will just store all Huey task signals into the database and display them in the Django admin.
pip install django-huey-monitor
INSTALLED_APPS = [
#...
'bx_django_utils', # https://github.com/boxine/bx_django_utils
'huey_monitor',
#...
]
Note: You must normally not change your Huey tasks.
Huey monitor model can store information about task hierarchy. But this information can't be collected automatically. You have to store these information in your Task code.
e.g.:
@task(context=True)
def sub_task(task, parent_task_id, chunk_data):
# Save the task hierarchy by:
TaskModel.objects.set_parent_task(
main_task_id=parent_task_id,
sub_task_id=task.id,
)
# ... do something with e.g.: chunk_data ...
@task(context=True)
def main_task(task):
for chunk_data in something:
sub_task(parent_task_id=task.id, chunk_data=chunk_data)
Working example can be found in the test app here: huey_monitor_tests/test_app/tasks.py
Running task can store progress information in a similar way as tqdm. So it's possible to see the progress in admin.
Minimal example:
@task(context=True)
def foobar_task(task, list_to_process):
process_info = ProcessInfo(
task,
desc='A description of this Job',
total=len(list_to_process)
)
for item in list_to_process:
# ...to something with the {item}...
process_info.update(n=1) # add the information that one item was processed
It is also possible to divide the work to several tasks and collect information about the processing of main-/sub-tasks.
Working example can be found in the test app here: huey_monitor_tests/test_app/tasks.py
It is possible to override list_filter of SignalInfoModelAdmin and TaskModelAdmin via settings.
e.g.:
HUEY_MONITOR_SIGNAL_INFO_MODEL_LIST_FILTER = ('task__name', 'signal_name')
HUEY_MONITOR_TASK_MODEL_LIST_FILTER = ('name', 'state__signal_name')
Note: This both settings are optional. In this example is the "hostname" filter not present ;)
Note: You can quickly test Huey Monitor with the test project, e.g:
~/django-huey-monitor$ ./manage.sh run_testserver
or in an isolated Docker container:
~/django-huey-monitor$ make up
More info see below.
(More Screenshots here: boxine.github.io/django-huey-monitor/)
To start developing e.g.:
~$ git clone https://github.com/boxine/django-huey-monitor.git
~$ cd django-huey-monitor
~/django-huey-monitor$ ./manage.py
~/django-huey-monitor$ make help
~/django-huey-monitor$ make up
Point our browser to: http://localhost:8000/
Our Makefile contains the following targets:
help List all commands
install install huey monitor package
update Update the dependencies as according to the pyproject.toml file
run-dev-server Run Django's developer server
test Run unittests
tox Run unittests via tox
makemessages Make and compile locales message files
clean Remove created files from the test project (e.g.: SQlite, static files)
build Update/Build docker services
up Start docker containers
down Stop all containers
shell-django go into a interactive bash shell in Django container
run-shell-django Build and start the Django container and go into shell
shell-huey1 go into a interactive bash shell in Huey worker container 1
shell-huey2 go into a interactive bash shell in Huey worker container 2
shell-huey3 go into a interactive bash shell in Huey worker container 3
shell-redis go into a interactive bash shell in Redis container
logs Display and follow docker logs
logs-django Display and follow docker logs only from "django" container
reload-django Reload the Django dev server
reload-huey Reload the Huey worker
restart Restart the containers
fire-test-tasks Call "fire-test-tasks" manage command to create some Huey Tasks
fire-many-test-tasks Call "fire-test-tasks" with --count 10000 to create many task entries ;)
fire-parallel-processing-task Just fire "parallel processing" Huey Task
delete-all-tasks-data Delete all Task/Signal database enties
It's also possible to run the test setup with SQLite and Huey immediate setup without docker:
~$ git clone https://github.com/boxine/django-huey-monitor.git
~$ cd django-huey-monitor
~/django-huey-monitor$ ./manage.py run_dev_server
| Huey Monitor | Django | Python |
|---|---|---|
| >v0.7.0 | v3.2, v4.1, v4.2 | v3.9, v3.10, v3.11 |
| >v0.6.0 | v3.2, v4.0, v4.1 | v3.9, v3.10, v3.11 |
| >v0.5.0 | v2.2, v3.1, v3.2 | v3.7, v3.8, v3.9 |
| <=v0.4.0 | v2.2, v3.0, v3.1 | v3.7, v3.8, v3.9 |
We refactor the project setup: Developer must reinit the repository.
Change CI and remove tests against Django 3.0, but add test run with Django v3.2
bx_py_utils was split and Django related stuff moved into: bx_django_utils
You must change your Django settings and replace the app name:
INSTALLED_APPS = [
#...
- 'bx_py_utils',
+ 'bx_django_utils',
'huey_monitor',
#...
]
ZeroDivisionError in adminlist_filter of SignalInfoModelAdmin and TaskModelAdmin via settingsmain branchpytest to normal unittestspoetry to pip-toolsProcessInfo.__init__ (contributed by Skrattoune)TaskModel.desc max. length and make SignalInfoModel.exception_line optionalTaskProgressModel and store progress information into TaskModelhuey_monitor.css filecumulate2parents flag to ProcessInfo (contributed by formacube)ProcessInfo.__str__()ProcessInfotox runs and use different Python versionsGPL. Patches welcome!
FAQs
Django based tool for monitoring huey task queue: https://github.com/coleifer/huey
We found that django-huey-monitor demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 4 open source maintainers collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
OpenSSF has published OSPS Baseline, an initiative designed to establish a minimum set of security-related best practices for open source software projects.
Security News
Michigan TypeScript founder Dimitri Mitropoulos implements WebAssembly runtime in TypeScript types, enabling Doom to run after processing 177 terabytes of type definitions.
Security News
vlt's new "reproduce" tool verifies npm packages against their source code, outperforming traditional provenance adoption in the JavaScript ecosystem.