Socket
Socket
Sign inDemoInstall

github.com/cdr/amboy

Package Overview
Dependencies
14
Maintainers
0
Alerts
File Explorer

Install Socket

Detect and block malicious and high-risk dependencies

Install

github.com/cdr/amboy

Package amboy provides basic infrastructure for running and describing tasks and task workflows with, potentially, minimal overhead and additional complexity. Amboy works with 4 basic logical objects: jobs, or descriptions of tasks; runnners, which are responsible for executing tasks; queues, that represent pipelines and offline workflows of tasks (e.g. not real time, processes that run outside of the primary execution path of a program); and dependencies that represent relationships between jobs. The inspiration for amboy was to be able to provide a unified way to define and run jobs, that would feel equally "native" for distributed applications and distributed web application, and move easily between different architectures. While amboy users will generally implement their own Job and dependency implementations, Amboy itself provides several example Queue implementations, as well as several generic examples and prototypes of Job and dependency.Manager objects. Generally speaking you should be able to use included amboy components to provide the queue and runner components, in conjunction with custom and generic job and dependency variations. Consider the following example: The amboy package proves a number of generic methods that, using the Queue.Stats() method, block until all jobs are complete. They provide different semantics, which may be useful in different circumstances. All of these functions wait until the total number of jobs submitted to the queue is equal to the number of completed jobs, and as a result these methods don't prevent other threads from adding jobs to the queue after beginning to wait. Additionally, there are a set of methods that allow callers to wait for a specific job to complete.


Version published
Maintainers
0

Readme

Source

=========================================================== amboy -- Task and Worker Pool Infrastructure |PkgGoDev|

.. |PkgGoDev| image:: https://pkg.go.dev/badge/cdr/amboy :target: https://pkg.go.dev/github.com/cdr/amboy

Overview

Amboy is a collection of interfaces and tools for running and managing asynchronous background work queues in the context of Go programs, and provides a number of interchangeable and robust methods for running tasks.

Features

Queues


Queue implementations impose ordering and dispatching behavior, and
describe the storage of tasks before and after work is
complete. Current queue implementations include:

- an ordered queue that dispatches tasks ordered by dependency
  information to ensure that dependent tasks that are completed before
  the tasks that depend on them.

- an unordered queue that ignores dependency information in tasks. For
  most basic cases these queues are ideal. (`LocalUnordered
  <https://godoc.org/github.com/cdr/amboy/queue#LocalUnordered>`_
  as implementation detail this queue dispatches tasks in a FIFO order.)

- a limited size queue that keep a fixed number of completed jobs in
  memory, which is ideal for long-running background processes.

- priority queues that dispatch tasks according to priority order.

- remote queues that store all tasks in an external storage system
  (e.g. a database) to support architectures where multiple processes
  can service the same underlying queue.

Queue Groups

The QueueGroup <https://godoc.org/github.com/cdr/amboy#QueueGroup>_ interface provides a mechanism to manage collections of queues. There are remote and local versions of the queue group possible, but these groups make it possible to create new queues at runtime, and improve the isolation of queues from each other.

Runners


Runners are the execution component of the worker pool, and are
embedded within the queues, and can be injected at run time before
starting the queue pool. The `LocalWorkers
<https://godoc.org/github.com/cdr/amboy/pool#LocalWorkers>`_
implementation executes tasks in a fixed-size worker pool, which is
the default of most queue implementations.

Additional implementation provide rate limiting, and it would be possible to
implement runners which used the REST interface to distribute workers to a
larger pool of processes, where existing runners simply use go routines.

Dependencies

The DependencyManager <https://godoc.org/github.com/cdr/amboy/dependency#Manager>_ interface makes it possible for tasks to express relationships to each other and to their environment so that Job operations can noop or block if their requirements are not satisfied. The data about relationships between jobs can inform task ordering as in the LocalOrdered <https://godoc.org/github.com/cdr/amboy/queue#LocalOrdered>_ queue.

The handling of dependency information is the responsibility of the queue implementation.

Management


The `management package
<https://godoc.org/github.com/cdr/amboy/management>`_ centers around a
`management interface
<https://godoc.org/github.com/cdr/amboy/management#Manager>`_ that provides
methods for reporting and safely interacting with the state of jobs.

REST Interface

The REST interface provides tools to submit jobs to an Amboy queue provided as a service. The rest package in Amboy provides the tools to build clients and services, although any client that can construct JSON formated Job object can use the REST API.

Additionally the REST package provides remote implementations of the management interface <https://godoc.org/github.com/cdr/amboy/rest#ManagementService>_ which makes it possible to manage and report on the jobs in an existing queue, and the abortable pool <https://godoc.org/github.com/cdr/amboy/rest#AbortablePoolManagementService>_ interface, that makes it possible to abort running jobs. These management tools can help administrators of larger amboy systems gain insights into the current behavior of the system, and promote safe and gentle operational interventions.

See the documentation of the REST package <https://godoc.org/github.com/cdr/amboy/rest>_

Logger


The Logger package provides amboy.Queue backed implementation of the
grip logging system's sender interface for asynchronous log message
delivery. These jobs do not support remote-backed queues.

Patterns
--------

The following patterns have emerged during our use of Amboy.

Base Job

Embed the job.Base <https://godoc.org/github.com/cdr/amboy/job/#Base>_ type in your amboy.Job implementations. This provides a number of helpers for basic job defintion in addition to implementations of all general methods in the interface. With the Base, you only need to implement a Run() method and whatever application logic is required for the task.

The only case where embedding the Base type may be contraindicated is in conjunction with the REST interface, as the Base type may require more complicated initialization processes.

Change Queue Implementations for Different Deployment Architectures


If your core application operations are implemented in terms of
amboy.Jobs, then you can: execute them independently of queues by
calling the ``Run()`` method, use a locally backed queue for
synchronous operation for short running queues, and use a limited size
queue or remote-backed queue as part of a long running service.

Please submit pull requests or `issues
<https://github.com/cdr/amboy>`_ with additional examples of amboy
use.

API and Documentation
---------------------

See the `godoc API documentation
<http://godoc.org/github.com/cdr/amboy>` for more information
about amboy interfaces and internals.

Development
-----------

Amboy is available for use under the terms of the Apache License (v2).

Issues
~~~~~~

If you encounter a problem with amboy, or would like to see a feature added,
please open an issue on the GitHub project!

Getting Started
~~~~~~~~~~~~~~~

All project automation is managed by a makefile, with all output captured in the
`build` directory. Consider the following operations: ::

   make build                   # runs a test compile
   make test                    # tests all packages
   make lint                    # lints all packages
   make test-<package>          # runs the tests only for a specific packages
   make lint-<package>          # lints a specific package
   make html-coverage-<package> # generates the coverage report for a specific package
   make coverage-html           # generates the coverage report for all packages

The buildsystem also has a number of flags, which may be useful for more
iterative development workflows: ::

  RUN_TEST=<TestName>   # specify a test name or regex to run a subset of tests
  RUN_COUNT=<num>       # run a test more than once to isolate an intermittent failure
  RACE_DETECTOR=true    # run specified tests with the race detector enabled.

Future Work
~~~~~~~~~~~

These features are speculative and there's not estimated time for
completion, but are provided here

- API Change: Remove or change the ``Jobs()`` and ``JobStats()`` methods on
  the queue so that they don't return channels. Use either iterators or
  provide other mechanisms for supporting the higher level functionality that
  these methods support.

- API Change: Replace the ``Runner()`` method on the queue interface with a
  ``Close()`` method.

- Feature: Add a queue implementation that job data jobs in a local,
  on-disk store, potentially using `badger
  <https://github.com/dgraph-io/badger>`_ for the backing store so jobs can
  persist between process starts without depending on MongoDB.

- Refactor: Simplify the MongoDB-based queues, to avoid the (now internal)
  driver interface.

FAQs

Last updated on 16 Feb 2021

Did you know?

Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.

Install

Related posts

SocketSocket SOC 2 Logo

Product

  • Package Alerts
  • Integrations
  • Docs
  • Pricing
  • FAQ
  • Roadmap

Stay in touch

Get open source security insights delivered straight into your inbox.


  • Terms
  • Privacy
  • Security

Made with ⚡️ by Socket Inc