github.com/cdr/amboy
Package amboy provides basic infrastructure for running and describing tasks and task workflows with, potentially, minimal overhead and additional complexity. Amboy works with 4 basic logical objects: jobs, or descriptions of tasks; runnners, which are responsible for executing tasks; queues, that represent pipelines and offline workflows of tasks (e.g. not real time, processes that run outside of the primary execution path of a program); and dependencies that represent relationships between jobs. The inspiration for amboy was to be able to provide a unified way to define and run jobs, that would feel equally "native" for distributed applications and distributed web application, and move easily between different architectures. While amboy users will generally implement their own Job and dependency implementations, Amboy itself provides several example Queue implementations, as well as several generic examples and prototypes of Job and dependency.Manager objects. Generally speaking you should be able to use included amboy components to provide the queue and runner components, in conjunction with custom and generic job and dependency variations. Consider the following example: The amboy package proves a number of generic methods that, using the Queue.Stats() method, block until all jobs are complete. They provide different semantics, which may be useful in different circumstances. All of these functions wait until the total number of jobs submitted to the queue is equal to the number of completed jobs, and as a result these methods don't prevent other threads from adding jobs to the queue after beginning to wait. Additionally, there are a set of methods that allow callers to wait for a specific job to complete.
Readme
amboy -- Task and Worker Pool Infrastructure |PkgGoDev|.. |PkgGoDev| image:: https://pkg.go.dev/badge/cdr/amboy :target: https://pkg.go.dev/github.com/cdr/amboy
Amboy is a collection of interfaces and tools for running and managing asynchronous background work queues in the context of Go programs, and provides a number of interchangeable and robust methods for running tasks.
Queues
Queue implementations impose ordering and dispatching behavior, and
describe the storage of tasks before and after work is
complete. Current queue implementations include:
- an ordered queue that dispatches tasks ordered by dependency
information to ensure that dependent tasks that are completed before
the tasks that depend on them.
- an unordered queue that ignores dependency information in tasks. For
most basic cases these queues are ideal. (`LocalUnordered
<https://godoc.org/github.com/cdr/amboy/queue#LocalUnordered>`_
as implementation detail this queue dispatches tasks in a FIFO order.)
- a limited size queue that keep a fixed number of completed jobs in
memory, which is ideal for long-running background processes.
- priority queues that dispatch tasks according to priority order.
- remote queues that store all tasks in an external storage system
(e.g. a database) to support architectures where multiple processes
can service the same underlying queue.
Queue Groups
The QueueGroup <https://godoc.org/github.com/cdr/amboy#QueueGroup>_
interface provides a mechanism to manage collections of queues. There are remote
and local versions of the queue group possible, but these groups make it
possible to create new queues at runtime, and improve the isolation of queues
from each other.
Runners
Runners are the execution component of the worker pool, and are
embedded within the queues, and can be injected at run time before
starting the queue pool. The `LocalWorkers
<https://godoc.org/github.com/cdr/amboy/pool#LocalWorkers>`_
implementation executes tasks in a fixed-size worker pool, which is
the default of most queue implementations.
Additional implementation provide rate limiting, and it would be possible to
implement runners which used the REST interface to distribute workers to a
larger pool of processes, where existing runners simply use go routines.
Dependencies
The DependencyManager <https://godoc.org/github.com/cdr/amboy/dependency#Manager>_
interface makes it possible for tasks to express relationships to each
other and to their environment so that Job operations can noop or
block if their requirements are not satisfied. The data about
relationships between jobs can inform task ordering as in the LocalOrdered <https://godoc.org/github.com/cdr/amboy/queue#LocalOrdered>_
queue.
The handling of dependency information is the responsibility of the queue implementation.
Management
The `management package
<https://godoc.org/github.com/cdr/amboy/management>`_ centers around a
`management interface
<https://godoc.org/github.com/cdr/amboy/management#Manager>`_ that provides
methods for reporting and safely interacting with the state of jobs.
REST Interface
The REST interface provides tools to submit jobs to an Amboy queue provided as a service. The rest package in Amboy provides the tools to build clients and services, although any client that can construct JSON formated Job object can use the REST API.
Additionally the REST package provides remote implementations of the management interface <https://godoc.org/github.com/cdr/amboy/rest#ManagementService>_
which makes it possible to manage and report on the jobs in an existing queue,
and the abortable pool <https://godoc.org/github.com/cdr/amboy/rest#AbortablePoolManagementService>_
interface, that makes it possible to abort running jobs. These management tools
can help administrators of larger amboy systems gain insights into the current
behavior of the system, and promote safe and gentle operational interventions.
See the documentation of the REST package <https://godoc.org/github.com/cdr/amboy/rest>_
Logger
The Logger package provides amboy.Queue backed implementation of the
grip logging system's sender interface for asynchronous log message
delivery. These jobs do not support remote-backed queues.
Patterns
--------
The following patterns have emerged during our use of Amboy.
Base Job
Embed the job.Base <https://godoc.org/github.com/cdr/amboy/job/#Base>_
type in your amboy.Job implementations. This provides a number of
helpers for basic job defintion in addition to implementations of all
general methods in the interface. With the Base, you only need to
implement a Run() method and whatever application logic is required
for the task.
The only case where embedding the Base type may be contraindicated is in conjunction with the REST interface, as the Base type may require more complicated initialization processes.
Change Queue Implementations for Different Deployment Architectures
If your core application operations are implemented in terms of
amboy.Jobs, then you can: execute them independently of queues by
calling the ``Run()`` method, use a locally backed queue for
synchronous operation for short running queues, and use a limited size
queue or remote-backed queue as part of a long running service.
Please submit pull requests or `issues
<https://github.com/cdr/amboy>`_ with additional examples of amboy
use.
API and Documentation
---------------------
See the `godoc API documentation
<http://godoc.org/github.com/cdr/amboy>` for more information
about amboy interfaces and internals.
Development
-----------
Amboy is available for use under the terms of the Apache License (v2).
Issues
~~~~~~
If you encounter a problem with amboy, or would like to see a feature added,
please open an issue on the GitHub project!
Getting Started
~~~~~~~~~~~~~~~
All project automation is managed by a makefile, with all output captured in the
`build` directory. Consider the following operations: ::
make build # runs a test compile
make test # tests all packages
make lint # lints all packages
make test-<package> # runs the tests only for a specific packages
make lint-<package> # lints a specific package
make html-coverage-<package> # generates the coverage report for a specific package
make coverage-html # generates the coverage report for all packages
The buildsystem also has a number of flags, which may be useful for more
iterative development workflows: ::
RUN_TEST=<TestName> # specify a test name or regex to run a subset of tests
RUN_COUNT=<num> # run a test more than once to isolate an intermittent failure
RACE_DETECTOR=true # run specified tests with the race detector enabled.
Future Work
~~~~~~~~~~~
These features are speculative and there's not estimated time for
completion, but are provided here
- API Change: Remove or change the ``Jobs()`` and ``JobStats()`` methods on
the queue so that they don't return channels. Use either iterators or
provide other mechanisms for supporting the higher level functionality that
these methods support.
- API Change: Replace the ``Runner()`` method on the queue interface with a
``Close()`` method.
- Feature: Add a queue implementation that job data jobs in a local,
on-disk store, potentially using `badger
<https://github.com/dgraph-io/badger>`_ for the backing store so jobs can
persist between process starts without depending on MongoDB.
- Refactor: Simplify the MongoDB-based queues, to avoid the (now internal)
driver interface.
FAQs
Package amboy provides basic infrastructure for running and describing tasks and task workflows with, potentially, minimal overhead and additional complexity. Amboy works with 4 basic logical objects: jobs, or descriptions of tasks; runnners, which are responsible for executing tasks; queues, that represent pipelines and offline workflows of tasks (e.g. not real time, processes that run outside of the primary execution path of a program); and dependencies that represent relationships between jobs. The inspiration for amboy was to be able to provide a unified way to define and run jobs, that would feel equally "native" for distributed applications and distributed web application, and move easily between different architectures. While amboy users will generally implement their own Job and dependency implementations, Amboy itself provides several example Queue implementations, as well as several generic examples and prototypes of Job and dependency.Manager objects. Generally speaking you should be able to use included amboy components to provide the queue and runner components, in conjunction with custom and generic job and dependency variations. Consider the following example: The amboy package proves a number of generic methods that, using the Queue.Stats() method, block until all jobs are complete. They provide different semantics, which may be useful in different circumstances. All of these functions wait until the total number of jobs submitted to the queue is equal to the number of completed jobs, and as a result these methods don't prevent other threads from adding jobs to the queue after beginning to wait. Additionally, there are a set of methods that allow callers to wait for a specific job to complete.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
A hospital in Mobile, Alabama, agreed to a settlement in a landmark ransomware death lawsuit, but is now reportedly reconsidering the agreement and refusing to pay.
Security News
A new report explores how advancements in LLMs are enhancing cyber threats, including polymorphic malware, personalized spearphishing, and the risk of hijacking customer service bots.
Security News
ESLint has approved an RFC that adds support for TypeScript configuration files, which is aimed at improving the developer experience and recognizing changes in the evolving JavaScript ecosystem.