github.com/tmobile/parallelizer
Package parallelizer is a library for enabling the addition of controlled parallelization utilizing a pool of worker goroutines in a simple manner. This is not intended as an external job queue, where outside programs may submit jobs, although it could easily be used to implement such a tool. The parallelizer package provides a Runner interface, which is for client applications to implement. Instances of the Runner interface may then be passed to the constructor functions NewSynchronousWorker or NewGoWorker, which construct objects conforming to the Worker interface. Data items may then be passed to the Worker instances via the Worker.Call method, and the processing completed and the final result obtained by calling Worker.Wait. A Runner implementation must provide a Runner.Run method, which will actually process the data in a goroutine and return a result; the result is then passed to the Runner.Integrate method, which is run synchronously with other Runner.Integrate calls, and which can submit additional data items for processing. Once all data is processed, and the client code has called Worker.Wait, the Worker will call the Runner.Result method to obtain the result. The Runner.Result method will be called exactly once; the returned value is cached in the Worker to be returned by future calls to Worker.Wait. The Worker.Call method may not be called again after Worker.Wait has been called. The parallelizer package also provides a Doer interface, which is for client applications to implement. Instances of the Doer interface may then be passed to the constructor function NewSerializer, which constructs objects conforming to the Serializer interface. Data items may then be passed to the Serializer objects to be executed via Doer.Do in a synchronous manner without necessarily blocking the calling goroutine. A Doer implementation must provide a Doer.Do method, which will actually process the data in a separate goroutine; each call will be executed synchronously, so thread-safety in Doer.Do is not a concern. When the code using the wrapping Serializer is done, it will call Serializer.Wait, which will call the Doer.Finish method and return its result to the caller. Note that none of the Serializer.Call methods may be called again after calling Serializer.Wait.
Readme
.. image:: https://img.shields.io/github/tag/tmobile/parallelizer.svg :target: https://github.com/tmobile/parallelizer/tags .. image:: https://img.shields.io/hexpm/l/plug.svg :target: https://github.com/tmobile/parallelizer/blob/master/LICENSE .. image:: https://travis-ci.org/tmobile/parallelizer.svg?branch=master :target: https://travis-ci.org/tmobile/parallelizer .. image:: https://coveralls.io/repos/github/tmobile/parallelizer/badge.svg?branch=master :target: https://coveralls.io/github/tmobile/parallelizer?branch=master .. image:: https://godoc.org/github.com/tmobile/parallelizer?status.svg :target: http://godoc.org/github.com/tmobile/parallelizer .. image:: https://img.shields.io/github/issues/tmobile/parallelizer.svg :target: https://github.com/tmobile/parallelizer/issues .. image:: https://img.shields.io/github/issues-pr/tmobile/parallelizer.svg :target: https://github.com/tmobile/parallelizer/pulls .. image:: https://goreportcard.com/badge/github.com/tmobile/parallelizer :target: https://goreportcard.com/report/github.com/tmobile/parallelizer
This repository contains Parallelizer. Parallelizer is a library for enabling the addition of controlled parallelization utilizing a pool of worker goroutines in a simple manner. This is not intended as an external job queue, where outside programs may submit jobs, although it could easily be used to implement such a tool.
The workers in this package implement the Worker interface,
consisting of two methods: Call() submits data to be worked on,
while Wait() stops data submission and waits for a final result.
Each worker is initialized with an instance of Runner provided by
the caller. A Runner must provide 3 methods: Run(), which
acts on the data and returns a result; Integrate(), which takes a
result produced by Run() and combines it with the other results
collected so far in an application-dependent fashion; and
Result(), which returns the final integrated result. Note that
each of these three methods may potentially run in a different
goroutine, and Run() may be invoked in parallel many times, but
calls to Integrate() are all performed synchronously, and
Result() is only called once after all calls to Run() and
Integrate() have completed. Further, it is not safe to call the
methods of the same Worker instance from any of the Runner
methods, but the Integrate() method will be called with an
instance of Worker that it is safe to call Call() on, allowing
recursion by having Run() return lists of additional data that
Integrate() then passes to Call().
In addition to the two interfaces above, aimed at allowing
parallelization, there are also three interfaces concerned with
serialization--having a bunch of disparate goroutines make calls that
are not thread-safe without necessarily blocking themselves. These
are the Doer interface, which is to be implemented by client code;
the Serializer interface, which allows those calls to be made; and
CallResult, which is a "future"-style object allowing asynchronous
retrieval of the results of a call made via Serializer. A user
would call Serializer.Call() (or any of its sister methods) to
have a Doer.Do() call made in the wrapped Doer in a
synchronized manner; then calling Serializer.Wait() will result in
a call to Doer.Finish(), returning the result.
The parallelizer package provides 3 implementations of the Worker
interface. The first is MockWorker, which is a simple struct that
may be used to mock the parallelizer out for the purposes of unit
testing.
The second implementation of Worker is a trivial implementation of
a synchronous worker, where all activity happens in a single
goroutine: the one that is calling the Call() and Wait()
methods. This worker is not thread-safe, and should not be used
simultaneously from different goroutines. An instance of a
synchronous worker may be created by passing the application's
Runner to the NewSynchronousWorker() function.
The third implementation of Worker is a parallel worker; this
worker starts up a defined number of worker goroutines, plus a manager
goroutine. This worker can be called into from almost any goroutine
(subject to the restriction noted above: Run(), Integrate(),
and Result() cannot invoke Call() on the worker itself, but
Integrate() is passed a variant that is safe to Call()). To
create an instance of a parallel worker, pass the runner and the
desired number of worker goroutines to the NewGoWorker() function.
The parallelizer package also provides 2 implementation of the
Serializer interface. The first is MockSerializer, which is a
simple struct that may be used to mock the serializer out for the
purposes of unit testing.
The second implementation of Serializer utilizes a manager
goroutine that actually makes the calls to the Doer.Do() method.
An instance of this serializer may be created by passing the
application's Doer to the NewSerializer() function.
The parallelizer package also provides a MockRunner, a struct
which implements the Runner interface. This may be useful for
other applications that utilize Runner, or which need to pass
Runner instances around internally. Similarly, it provides the
MockDoer, another struct which implements the Doer interface,
and MockCallResult, which implements the CallResult interface.
This latter may be useful if the application being tested uses
Serializer.CallAsync().
This repository is a standard go repository, and so may be tested and
built in the standard go ways. However, the repository also contains
a Makefile to aid in repeatable testing and reformatting;
developers that wish to contribute to Parallelizer may find it useful
to utilize make to ensure that their code conforms to the
standards enforced by Travis CI. The following is a run-down of the
available make targets.
make format-testThis target is called by Travis to ensure that the formatting conforms
to that recommended by the standard go tools goimports and
gofmt. Most developers should prefer the make format target,
which is automatically run by make test or make cover, and
will rewrite non-conforming files. Note that goimports is a
third-party package; it may be installed using::
% go get -u -v golang.org/x/tools/cmd/goimports
make formatThis target may be called by developers to ensure that the source code
conforms to the recommended style. It runs goimports and
gofmt to this end. Most developers will prefer to use make test or make cover, which automatically invoke make format.
Note that goimports is a third-party package; it may be installed
using::
% go get -u -v golang.org/x/tools/cmd/goimports
make lintThis target may be called to run a lint check. This tests for such
things as the presence of documentation comments on exported functions
and types, etc. To this end, this target runs golint in enforcing
mode. Most developers will prefer to use make test or make cover, which automatically invoke make lint. Note that
golint is a third-party package; it may be installed using::
% go get -u -v golang.org/x/lint/golint
make vetThis target may be called to run a "vet" check. This vets the source
code, looking for common problems prior to attempting to compile it.
Most developers will prefer to use make test or make cover,
which automatically invoke make vet.
make test-onlyThis target may be called to run only the unit tests. A coverage
profile will be output to coverage.out, but none of the other
tests, such as make vet, will be invoked. Most developers will
prefer to use make test or make cover, which automatically
invoke make test-only, among other targets.
make testThis target may be called to run all the tests. It ensures that
make format, make lint, make vet, and make test-only
are all called, in that order.
make coverThis target may be called to run make test, but will additionally
generate an HTML file named coverage.html which will report on the
coverage of the source code by the test suite.
make cleanThis target may be called to remove the temporary files
coverage.out and coverage.html, as well as any future
temporary files that are added in the testing process.
Contributions are welcome! Please ensure that all tests described above pass prior to proposing pull requests; pull requests that do not pass the test suite unfortunately cannot be merged. Also, please ensure adequate test coverage of additional code and branches of existing code; the ideal target is 100% coverage, to ensure adequate confidence in the function of Parallelizer.
FAQs
Package parallelizer is a library for enabling the addition of controlled parallelization utilizing a pool of worker goroutines in a simple manner. This is not intended as an external job queue, where outside programs may submit jobs, although it could easily be used to implement such a tool. The parallelizer package provides a Runner interface, which is for client applications to implement. Instances of the Runner interface may then be passed to the constructor functions NewSynchronousWorker or NewGoWorker, which construct objects conforming to the Worker interface. Data items may then be passed to the Worker instances via the Worker.Call method, and the processing completed and the final result obtained by calling Worker.Wait. A Runner implementation must provide a Runner.Run method, which will actually process the data in a goroutine and return a result; the result is then passed to the Runner.Integrate method, which is run synchronously with other Runner.Integrate calls, and which can submit additional data items for processing. Once all data is processed, and the client code has called Worker.Wait, the Worker will call the Runner.Result method to obtain the result. The Runner.Result method will be called exactly once; the returned value is cached in the Worker to be returned by future calls to Worker.Wait. The Worker.Call method may not be called again after Worker.Wait has been called. The parallelizer package also provides a Doer interface, which is for client applications to implement. Instances of the Doer interface may then be passed to the constructor function NewSerializer, which constructs objects conforming to the Serializer interface. Data items may then be passed to the Serializer objects to be executed via Doer.Do in a synchronous manner without necessarily blocking the calling goroutine. A Doer implementation must provide a Doer.Do method, which will actually process the data in a separate goroutine; each call will be executed synchronously, so thread-safety in Doer.Do is not a concern. When the code using the wrapping Serializer is done, it will call Serializer.Wait, which will call the Doer.Finish method and return its result to the caller. Note that none of the Serializer.Call methods may be called again after calling Serializer.Wait.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
The Python Software Foundation has secured a 5-year sponsorship from Fastly that supports PSF's activities and events, most notably the security and reliability of the Python Package Index (PyPI).
Security News
LDAPjs, an LDAP Client and Server API for Node.js, was decommissioned after its maintainer received an abusive email from a user, raising concerns about this form of abuse as a potential attack vector.
Security News
CISA launched a new project called Vulnrichment to enrich CVEs with details that help prioritize patching and mitigation efforts, as the NVD backlog of unenriched CVEs awaiting analysis surpasses 10,000.