Interfaces and shared infrastructure for generic task processing (also known as taskproc) at Yelp.
hello-world.py is a very simple annotated example that launches a task to echo hello world. From the root of the repository, run:
docker-compose -f examples/cluster/docker-compose.yaml \
run playground examples/hello-world.py
This will bring up a single master, single agent Mesos cluster using Docker Compose and launch a single task which will print "hello world" to the sandbox's stdout before terminating.
Other examples available include:
async.py Example of the async task runner.
dynamo_persistence.py
Example that shows how task events may be persisted to DynamoDB using the stateful plugin.
file_persistence.py
Example that shows how task events may be persisted to disk using the stateful plugin.
promise.py Example that shows how the promise/future task runner (not yet implemented) may be used.
subscription.py Example of the subscription task runner.
sync.py Brief example using the sync task runner.
timeout.py
Example that shows how to timeout a task execution using the timeout plugin.
retry.py
Example that shows how to retry a task on failure using the retry plugin.
task_logging.py
Example that shows how to fetch task logs from Mesos agents using the logging plugin.
From the root of the repository, run:
make
Plugins can be chained to create a task execution pipeline with more than one property. Please refer to persistence/retry/timeout examples.
Implements all required interfaces to talk to Mesos deployment. This plugin uses PyMesos to communicate with Mesos.
Implements an executor to timeout task execution.
Implements an executor to retry task execution upon failure.
Implements an executor to retrieve task logs from Mesos agents. Note that it has to be the immediate upstream executor of the mesos executor.
Event objectsTODO: documentation
Runners provide specific concurrency semantics and are supposed to be platform independent.
Running a task is a blocking operation. sync runners block until the running task has completed or a stop event is received.
Provide callbacks for different events in tasks' lifecycle. async runners allow tasks to specify one or more EventHandlers which consist of predicates and callbacks. Predicates are evaluated when an update is received from the task (e.g. that it has terminated and whether or not it has succeded) and if the predicate passes, the callback is called.
Running a task returns future object.
Provide a queue object and receive all events in there.
FAQs
Framework for task processing executors and configuration
We found that task-processing demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Research
Security News
Attackers used a malicious npm package typosquatting a popular ESLint plugin to steal sensitive data, execute commands, and exploit developer systems.
Security News
The Ultralytics' PyPI Package was compromised four times in one weekend through GitHub Actions cache poisoning and failure to rotate previously compromised API tokens.
Security News
Research
Socket researchers found a malicious Maven package impersonating the legitimate ‘XZ for Java’ library, introducing a backdoor for remote code execution.