|GHAction| |CircleCI| |codecov|
|Pydralogo|
.. |Pydralogo| image:: https://raw.githubusercontent.com/nipype/pydra/master/docs/logo/pydra_logo.jpg
:width: 200px
:alt: pydra logo
.. |GHAction| image:: https://github.com/nipype/pydra/workflows/Pydra/badge.svg
:alt: GitHub Actions CI
:target: https://github.com/nipype/Pydra/actions
.. |CircleCI| image:: https://circleci.com/gh/nipype/pydra.svg?style=svg
:alt: CircleCI
.. |codecov| image:: https://codecov.io/gh/nipype/pydra/branch/master/graph/badge.svg
:alt: codecov
======================
Pydra: Dataflow Engine
A simple dataflow engine with scalable semantics.
Pydra is a rewrite of the Nipype engine with mapping and joining as
first-class operations. It forms the core of the Nipype 2.0 ecosystem.
The goal of pydra is to provide a lightweight Python dataflow engine for DAG
construction, manipulation, and distributed execution.
Feature list:
- Python 3.7+ using type annotation and
attrs <https://www.attrs.org/en/stable/>
_ - Composable dataflows with simple node semantics. A dataflow can be a node of another dataflow.
splitter
and combiner
provides many ways of compressing complex loop semantics- Cached execution with support for a global cache across dataflows and users
- Distributed execution, presently via ConcurrentFutures, SLURM, and Dask (this is an experimental implementation with limited testing)
API Documentation <https://nipype.github.io/pydra/>
_
Learn more about Pydra
SciPy 2020 Proceedings <http://conference.scipy.org/proceedings/scipy2020/pydra.html>
_PyCon 2020 Poster <https://docs.google.com/presentation/d/10tS2I34rS0G9qz6v29qVd77OUydjP_FdBklrgAGmYSw/edit?usp=sharing>
_Explore Pydra interactively <https://github.com/nipype/pydra-tutorial>
_ (the tutorial can be also run using Binder service)
|Binder|
.. |Binder| image:: https://mybinder.org/badge_logo.svg
:alt: Binder
Please note that mybinder times out after an hour.
Installation
::
pip install pydra
Note that installation fails with older versions of pip on Windows. Upgrade pip before installing:
::
pip install –upgrade pip
pip install pydra
Developer installation
Pydra requires Python 3.7+. To install in developer mode:
::
git clone git@github.com:nipype/pydra.git
cd pydra
pip install -e ".[dev]"
In order to run pydra's test locally:
::
pytest -vs pydra
If you want to test execution with Dask:
::
git clone git@github.com:nipype/pydra.git
cd pydra
pip install -e ".[dask]"
It is also useful to install pre-commit:
::
pip install pre-commit
pre-commit