pipen

A pipeline framework for python

Documentation | ChangeLog | Examples | API

Features

• Easy to use
• Nearly zero-configuration
• Nice logging
• Highly extendable
• Cloud support naively

Installation

pip install -U pipen

Quickstart

example.py

from pipen import Proc, Pipen, run

class P1(Proc):
    """Sort input file"""
    input = "infile"
    input_data = ["/tmp/data.txt"]
    output = "outfile:file:intermediate.txt"
    script = "cat {{in.infile}} | sort > {{out.outfile}}"

class P2(Proc):
    """Paste line number"""
    requires = P1
    input = "infile:file"
    output = "outfile:file:result.txt"
    script = "paste <(seq 1 3) {{in.infile}} > {{out.outfile}}"

# class MyPipeline(Pipen):
#     starts = P1

if __name__ == "__main__":
    # MyPipeline().run()
    run("MyPipeline", starts=P1)

> echo -e "3\n2\n1" > /tmp/data.txt
> python example.py

04-17 16:19:35 I core                   _____________________________________   __
04-17 16:19:35 I core                   ___  __ \___  _/__  __ \__  ____/__  | / /
04-17 16:19:35 I core                   __  /_/ /__  / __  /_/ /_  __/  __   |/ /
04-17 16:19:35 I core                   _  ____/__/ /  _  ____/_  /___  _  /|  /
04-17 16:19:35 I core                   /_/     /___/  /_/     /_____/  /_/ |_/
04-17 16:19:35 I core
04-17 16:19:35 I core                               version: 0.17.3
04-17 16:19:35 I core
04-17 16:19:35 I core    ╔═══════════════════════════ MYPIPELINE ════════════════════════════╗
04-17 16:19:35 I core    ║ My pipeline                                                       ║
04-17 16:19:35 I core    ╚═══════════════════════════════════════════════════════════════════╝
04-17 16:19:35 I core    plugins         : verbose v0.14.1
04-17 16:19:35 I core    # procs         : 2
04-17 16:19:35 I core    profile         : default
04-17 16:19:35 I core    outdir          :
                /home/pwwang/github/pipen/examples/MyPipeline-output
04-17 16:19:35 I core    cache           : True
04-17 16:19:35 I core    dirsig          : 1
04-17 16:19:35 I core    error_strategy  : ignore
04-17 16:19:35 I core    forks           : 1
04-17 16:19:35 I core    lang            : bash
04-17 16:19:35 I core    loglevel        : info
04-17 16:19:35 I core    num_retries     : 3
04-17 16:19:35 I core    scheduler       : local
04-17 16:19:35 I core    submission_batch: 8
04-17 16:19:35 I core    template        : liquid
04-17 16:19:35 I core    workdir         :
                 /home/pwwang/github/pipen/examples/.pipen/MyPipeline
04-17 16:19:35 I core    plugin_opts     :
04-17 16:19:35 I core    template_opts   : filters={'realpath': <function realpath at
                 0x7fc3eba12...
04-17 16:19:35 I core                    : globals={'realpath': <function realpath at
                 0x7fc3eba12...
04-17 16:19:35 I core    Initializing plugins ...
04-17 16:19:36 I core
04-17 16:19:36 I core    ╭─────────────────────────────── P1 ────────────────────────────────╮
04-17 16:19:36 I core    │ Sort input file                                                   │
04-17 16:19:36 I core    ╰───────────────────────────────────────────────────────────────────╯
04-17 16:19:36 I core    P1: Workdir:
                 '/home/pwwang/github/pipen/examples/.pipen/MyPipeline/P1'
04-17 16:19:36 I core    P1: <<< [START]
04-17 16:19:36 I core    P1: >>> ['P2']
04-17 16:19:36 I verbose P1: in.infile: /tmp/data.txt
04-17 16:19:36 I verbose P1: out.outfile:
                 /home/pwwang/github/pipen/examples/.pipen/MyPipeline/P1/0/output/intermediate
                 .txt
04-17 16:19:38 I verbose P1: Time elapsed: 00:00:02.051s
04-17 16:19:38 I core
04-17 16:19:38 I core    ╭═══════════════════════════════ P2 ════════════════════════════════╮
04-17 16:19:38 I core    ║ Paste line number                                                 ║
04-17 16:19:38 I core    ╰═══════════════════════════════════════════════════════════════════╯
04-17 16:19:38 I core    P2: Workdir:
                 '/home/pwwang/github/pipen/examples/.pipen/MyPipeline/P2'
04-17 16:19:38 I core    P2: <<< ['P1']
04-17 16:19:38 I core    P2: >>> [END]
04-17 16:19:38 I verbose P2: in.infile:
                 /home/pwwang/github/pipen/examples/.pipen/MyPipeline/P1/0/output/intermediate
                 .txt
04-17 16:19:38 I verbose P2: out.outfile:
                 /home/pwwang/github/pipen/examples/MyPipeline-output/P2/result.txt
04-17 16:19:41 I verbose P2: Time elapsed: 00:00:02.051s
04-17 16:19:41 I core


              MYPIPELINE: 100%|██████████████████████████████| 2/2 [00:06<00:00, 0.35 procs/s]

> cat ./MyPipeline-output/P2/result.txt
1       1
2       2
3       3

Examples

See more examples at examples/ and a more realcase example at:

https://github.com/pwwang/pipen-report/tree/master/example

Plugin gallery

Plugins make pipen even better.

• pipen-annotate: Use docstring to annotate pipen processes
• pipen-args: Command line argument parser for pipen
• pipen-board: Visualize configuration and running of pipen pipelines on the web
• pipen-diagram: Draw pipeline diagrams for pipen
• pipen-dry: Dry runner for pipen pipelines
• pipen-filters: Add a set of useful filters for pipen templates.
• pipen-lock: Process lock for pipen to prevent multiple runs at the same time.
• pipen-log2file: Save running logs to file for pipen
• pipen-poplog: Populate logs from jobs to running log of the pipeline
• pipen-report: Generate report for pipen
• pipen-runinfo: Save running information to file for pipen
• pipen-verbose: Add verbosal information in logs for pipen.
• pipen-gcs: A plugin for pipen to handle files in Google Cloud Storage.
• pipen-cli-init: A pipen CLI plugin to create a pipen project (pipeline)
• pipen-cli-ref: Make reference documentation for processes
• pipen-cli-require: A pipen cli plugin check the requirements of a pipeline
• pipen-cli-run: A pipen cli plugin to run a process or a pipeline