Sign inDemoInstall

Package Overview
File Explorer

Install Socket

Protect your apps from supply chain attacks


Package amboy provides basic infrastructure for running and describing tasks and task workflows with, potentially, minimal overhead and additional complexity. Amboy works with 4 basic logical objects: jobs, or descriptions of tasks; runnners, which are responsible for executing tasks; queues, that represent pipelines and offline workflows of tasks (e.g. not real time, processes that run outside of the primary execution path of a program); and dependencies that represent relationships between jobs. The inspiration for amboy was to be able to provide a unified way to define and run jobs, that would feel equally "native" for distributed applications and distributed web application, and move easily between different architectures. While amboy users will generally implement their own Job and dependency implementations, Amboy itself provides several example Queue implementations, as well as several generic examples and prototypes of Job and dependency.Manager objects. Generally speaking you should be able to use included amboy components to provide the queue and runner components, in conjunction with custom and generic job and dependency variations. Consider the following example: The amboy package proves a number of generic methods that, using the Queue.Stats() method, block until all jobs are complete. They provide different semantics, which may be useful in different circumstances. All of these functions wait until the total number of jobs submitted to the queue is equal to the number of completed jobs, and as a result these methods don't prevent other threads from adding jobs to the queue after beginning to wait. Additionally, there are a set of methods that allow callers to wait for a specific job to complete.


Version published


``amboy`` -- Task and Worker Pool Infrastructure |PkgGoDev|

.. |PkgGoDev| image::


Amboy is a collection of interfaces and tools for running and managing
asynchronous background work queues in the context of Go programs, and
provides a number of interchangeable and robust methods for running



Queue implementations impose ordering and dispatching behavior, and
describe the storage of tasks before and after work is
complete. Current queue implementations include:

- an ordered queue that dispatches tasks ordered by dependency
  information to ensure that dependent tasks that are completed before
  the tasks that depend on them.

- an unordered queue that ignores dependency information in tasks. For
  most basic cases these queues are ideal. (`LocalUnordered
  as implementation detail this queue dispatches tasks in a FIFO order.)

- a limited size queue that keep a fixed number of completed jobs in
  memory, which is ideal for long-running background processes.

- priority queues that dispatch tasks according to priority order.

- remote queues that store all tasks in an external storage system
  (e.g. a database) to support architectures where multiple processes
  can service the same underlying queue.

Queue Groups

The `QueueGroup <>`_
interface provides a mechanism to manage collections of queues. There are remote
and local versions of the queue group possible, but these groups make it
possible to create new queues at runtime, and improve the isolation of queues
from each other.


Runners are the execution component of the worker pool, and are
embedded within the queues, and can be injected at run time before
starting the queue pool. The `LocalWorkers
implementation executes tasks in a fixed-size worker pool, which is
the default of most queue implementations.

Additional implementation provide rate limiting, and it would be possible to
implement runners which used the REST interface to distribute workers to a
larger pool of processes, where existing runners simply use go routines.


The `DependencyManager
interface makes it possible for tasks to express relationships to each
other and to their environment so that Job operations can noop or
block if their requirements are not satisfied. The data about
relationships between jobs can inform task ordering as in the `LocalOrdered

The handling of dependency information is the responsibility of the
queue implementation.


The `management package
<>`_ centers around a
`management interface
<>`_ that provides
methods for reporting and safely interacting with the state of jobs.

REST Interface

The REST interface provides tools to submit jobs to an Amboy queue
provided as a service. The rest package in Amboy provides the tools to
build clients and services, although any client that can construct
JSON formated Job object can use the REST API.

Additionally the REST package provides remote implementations of the `management
interface <>`_
which makes it possible to manage and report on the jobs in an existing queue,
and the `abortable pool
interface, that makes it possible to abort running jobs. These management tools
can help administrators of larger amboy systems gain insights into the current
behavior of the system, and promote safe and gentle operational interventions.

See the documentation of the `REST package


The Logger package provides amboy.Queue backed implementation of the
grip logging system's sender interface for asynchronous log message
delivery. These jobs do not support remote-backed queues.


The following patterns have emerged during our use of Amboy.

Base Job

Embed the `job.Base
type in your amboy.Job implementations. This provides a number of
helpers for basic job defintion in addition to implementations of all
general methods in the interface. With the Base, you only need to
implement a ``Run()`` method and whatever application logic is required
for the task.

The only case where embedding the Base type *may* be contraindicated is
in conjunction with the REST interface, as the Base type may require
more complicated initialization processes.

Change Queue Implementations for Different Deployment Architectures

If your core application operations are implemented in terms of
amboy.Jobs, then you can: execute them independently of queues by
calling the ``Run()`` method, use a locally backed queue for
synchronous operation for short running queues, and use a limited size
queue or remote-backed queue as part of a long running service.

Please submit pull requests or `issues
<>`_ with additional examples of amboy

API and Documentation

See the `godoc API documentation
<>` for more information
about amboy interfaces and internals.


Amboy is available for use under the terms of the Apache License (v2).


If you encounter a problem with amboy, or would like to see a feature added,
please open an issue on the GitHub project!

Getting Started

All project automation is managed by a makefile, with all output captured in the
`build` directory. Consider the following operations: ::

   make build                   # runs a test compile
   make test                    # tests all packages
   make lint                    # lints all packages
   make test-<package>          # runs the tests only for a specific packages
   make lint-<package>          # lints a specific package
   make html-coverage-<package> # generates the coverage report for a specific package
   make coverage-html           # generates the coverage report for all packages

The buildsystem also has a number of flags, which may be useful for more
iterative development workflows: ::

  RUN_TEST=<TestName>   # specify a test name or regex to run a subset of tests
  RUN_COUNT=<num>       # run a test more than once to isolate an intermittent failure
  RACE_DETECTOR=true    # run specified tests with the race detector enabled.

Future Work

These features are speculative and there's not estimated time for
completion, but are provided here

- API Change: Remove or change the ``Jobs()`` and ``JobStats()`` methods on
  the queue so that they don't return channels. Use either iterators or
  provide other mechanisms for supporting the higher level functionality that
  these methods support.

- API Change: Replace the ``Runner()`` method on the queue interface with a
  ``Close()`` method.

- Feature: Add a queue implementation that job data jobs in a local,
  on-disk store, potentially using `badger
  <>`_ for the backing store so jobs can
  persist between process starts without depending on MongoDB.

- Refactor: Simplify the MongoDB-based queues, to avoid the (now internal)
  driver interface.


Last updated on 16 Feb 2021

Did you know?

Socket installs a GitHub app to automatically flag issues on every pull request and report the health of your dependencies. Find out what is inside your node modules and prevent malicious activity before you update the dependencies.


Related posts

SocketSocket SOC 2 Logo


  • Package Alerts
  • Integrations
  • Docs
  • Pricing
  • FAQ
  • Roadmap

Stay in touch

Get open source security insights delivered straight into your inbox.

  • Terms
  • Privacy
  • Security

Made with ⚡️ by Socket Inc