Socket
Socket
Sign inDemoInstall

github.com/cdr/amboy

Package Overview
Dependencies
14
Maintainers
0
Alerts
File Explorer

Install Socket

Protect your apps from supply chain attacks

Install

github.com/cdr/amboy

Package amboy provides basic infrastructure for running and describing tasks and task workflows with, potentially, minimal overhead and additional complexity. Amboy works with 4 basic logical objects: jobs, or descriptions of tasks; runnners, which are responsible for executing tasks; queues, that represent pipelines and offline workflows of tasks (e.g. not real time, processes that run outside of the primary execution path of a program); and dependencies that represent relationships between jobs. The inspiration for amboy was to be able to provide a unified way to define and run jobs, that would feel equally "native" for distributed applications and distributed web application, and move easily between different architectures. While amboy users will generally implement their own Job and dependency implementations, Amboy itself provides several example Queue implementations, as well as several generic examples and prototypes of Job and dependency.Manager objects. Generally speaking you should be able to use included amboy components to provide the queue and runner components, in conjunction with custom and generic job and dependency variations. Consider the following example: The amboy package proves a number of generic methods that, using the Queue.Stats() method, block until all jobs are complete. They provide different semantics, which may be useful in different circumstances. All of these functions wait until the total number of jobs submitted to the queue is equal to the number of completed jobs, and as a result these methods don't prevent other threads from adding jobs to the queue after beginning to wait. Additionally, there are a set of methods that allow callers to wait for a specific job to complete.

    v0.0.0-20210216153014-b4d1232640b9

Version published
Maintainers
0

Readme

===========================================================
``amboy`` -- Task and Worker Pool Infrastructure |PkgGoDev|
===========================================================

.. |PkgGoDev| image:: https://pkg.go.dev/badge/cdr/amboy
	      :target: https://pkg.go.dev/github.com/cdr/amboy

Overview
--------

Amboy is a collection of interfaces and tools for running and managing
asynchronous background work queues in the context of Go programs, and
provides a number of interchangeable and robust methods for running
tasks.


Features
--------

Queues
~~~~~~

Queue implementations impose ordering and dispatching behavior, and
describe the storage of tasks before and after work is
complete. Current queue implementations include:

- an ordered queue that dispatches tasks ordered by dependency
  information to ensure that dependent tasks that are completed before
  the tasks that depend on them.

- an unordered queue that ignores dependency information in tasks. For
  most basic cases these queues are ideal. (`LocalUnordered
  <https://godoc.org/github.com/cdr/amboy/queue#LocalUnordered>`_
  as implementation detail this queue dispatches tasks in a FIFO order.)

- a limited size queue that keep a fixed number of completed jobs in
  memory, which is ideal for long-running background processes.

- priority queues that dispatch tasks according to priority order.

- remote queues that store all tasks in an external storage system
  (e.g. a database) to support architectures where multiple processes
  can service the same underlying queue.

Queue Groups
~~~~~~~~~~~~

The `QueueGroup <https://godoc.org/github.com/cdr/amboy#QueueGroup>`_
interface provides a mechanism to manage collections of queues. There are remote
and local versions of the queue group possible, but these groups make it
possible to create new queues at runtime, and improve the isolation of queues
from each other.

Runners
~~~~~~~

Runners are the execution component of the worker pool, and are
embedded within the queues, and can be injected at run time before
starting the queue pool. The `LocalWorkers
<https://godoc.org/github.com/cdr/amboy/pool#LocalWorkers>`_
implementation executes tasks in a fixed-size worker pool, which is
the default of most queue implementations.

Additional implementation provide rate limiting, and it would be possible to
implement runners which used the REST interface to distribute workers to a
larger pool of processes, where existing runners simply use go routines.

Dependencies
~~~~~~~~~~~~

The `DependencyManager
<https://godoc.org/github.com/cdr/amboy/dependency#Manager>`_
interface makes it possible for tasks to express relationships to each
other and to their environment so that Job operations can noop or
block if their requirements are not satisfied. The data about
relationships between jobs can inform task ordering as in the `LocalOrdered
<https://godoc.org/github.com/cdr/amboy/queue#LocalOrdered>`_
queue.

The handling of dependency information is the responsibility of the
queue implementation.

Management
~~~~~~~~~~

The `management package
<https://godoc.org/github.com/cdr/amboy/management>`_ centers around a
`management interface
<https://godoc.org/github.com/cdr/amboy/management#Manager>`_ that provides
methods for reporting and safely interacting with the state of jobs.

REST Interface
~~~~~~~~~~~~~~

The REST interface provides tools to submit jobs to an Amboy queue
provided as a service. The rest package in Amboy provides the tools to
build clients and services, although any client that can construct
JSON formated Job object can use the REST API.

Additionally the REST package provides remote implementations of the `management
interface <https://godoc.org/github.com/cdr/amboy/rest#ManagementService>`_
which makes it possible to manage and report on the jobs in an existing queue,
and the `abortable pool
<https://godoc.org/github.com/cdr/amboy/rest#AbortablePoolManagementService>`_
interface, that makes it possible to abort running jobs. These management tools
can help administrators of larger amboy systems gain insights into the current
behavior of the system, and promote safe and gentle operational interventions.

See the documentation of the `REST package
<https://godoc.org/github.com/cdr/amboy/rest>`_

Logger
~~~~~~

The Logger package provides amboy.Queue backed implementation of the
grip logging system's sender interface for asynchronous log message
delivery. These jobs do not support remote-backed queues.

Patterns
--------

The following patterns have emerged during our use of Amboy.

Base Job
~~~~~~~~

Embed the `job.Base
<https://godoc.org/github.com/cdr/amboy/job/#Base>`_
type in your amboy.Job implementations. This provides a number of
helpers for basic job defintion in addition to implementations of all
general methods in the interface. With the Base, you only need to
implement a ``Run()`` method and whatever application logic is required
for the task.

The only case where embedding the Base type *may* be contraindicated is
in conjunction with the REST interface, as the Base type may require
more complicated initialization processes.

Change Queue Implementations for Different Deployment Architectures
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

If your core application operations are implemented in terms of
amboy.Jobs, then you can: execute them independently of queues by
calling the ``Run()`` method, use a locally backed queue for
synchronous operation for short running queues, and use a limited size
queue or remote-backed queue as part of a long running service.

Please submit pull requests or `issues
<https://github.com/cdr/amboy>`_ with additional examples of amboy
use.

API and Documentation
---------------------

See the `godoc API documentation
<http://godoc.org/github.com/cdr/amboy>` for more information
about amboy interfaces and internals.

Development
-----------

Amboy is available for use under the terms of the Apache License (v2).

Issues
~~~~~~

If you encounter a problem with amboy, or would like to see a feature added,
please open an issue on the GitHub project!

Getting Started
~~~~~~~~~~~~~~~

All project automation is managed by a makefile, with all output captured in the
`build` directory. Consider the following operations: ::

   make build                   # runs a test compile
   make test                    # tests all packages
   make lint                    # lints all packages
   make test-<package>          # runs the tests only for a specific packages
   make lint-<package>          # lints a specific package
   make html-coverage-<package> # generates the coverage report for a specific package
   make coverage-html           # generates the coverage report for all packages

The buildsystem also has a number of flags, which may be useful for more
iterative development workflows: ::

  RUN_TEST=<TestName>   # specify a test name or regex to run a subset of tests
  RUN_COUNT=<num>       # run a test more than once to isolate an intermittent failure
  RACE_DETECTOR=true    # run specified tests with the race detector enabled.

Future Work
~~~~~~~~~~~

These features are speculative and there's not estimated time for
completion, but are provided here

- API Change: Remove or change the ``Jobs()`` and ``JobStats()`` methods on
  the queue so that they don't return channels. Use either iterators or
  provide other mechanisms for supporting the higher level functionality that
  these methods support.

- API Change: Replace the ``Runner()`` method on the queue interface with a
  ``Close()`` method.

- Feature: Add a queue implementation that job data jobs in a local,
  on-disk store, potentially using `badger
  <https://github.com/dgraph-io/badger>`_ for the backing store so jobs can
  persist between process starts without depending on MongoDB.

- Refactor: Simplify the MongoDB-based queues, to avoid the (now internal)
  driver interface.

FAQs

Last updated on 16 Feb 2021

Did you know?

Socket installs a GitHub app to automatically flag issues on every pull request and report the health of your dependencies. Find out what is inside your node modules and prevent malicious activity before you update the dependencies.

Install

Related posts

SocketSocket SOC 2 Logo

Product

  • Package Alerts
  • Integrations
  • Docs
  • Pricing
  • FAQ
  • Roadmap

Stay in touch

Get open source security insights delivered straight into your inbox.


  • Terms
  • Privacy
  • Security

Made with ⚡️ by Socket Inc