airflow-windmill

Windmill

Drag'n'drop web app to manage and create Airflow DAGs. DAGs are described usinga JSON "wml" file, which can be transpiled into a Python DAG file and pushed to a configured git repository.

• Front end is built using React on Typescript
• Back end is built using Flask on Python 3.6+

Getting Started

• Install with pip install airflow-windmill
  • Airflow is expected to be installed on the system. This allows Windmill to run with arbitrary versions of Airflow
  • Otherwise it can be packaged with windmill using pip install airflow-windmill[airflow]. The version is defined in pyproject.toml
• Run windmill init to create a local Windmill project
• cd windmill-project
• Run windmill run from this folder to run the app locally

MVP Required Features

Front-End Features

• Dynamic Operators
• Menu Dropdowns
• Load Operators from App
• Format operator display into classes
• Search functionality for operators
• Basic operator level properties
• Implement DAG level properties
• New DAG Functionality
• Parameter Tooltips
• Render arbitrary viewport windows for New/Save/Load etc
• Overwrite/Save prompt on New
• DAG renaming and save functionality
• Open dag from menu
• Make save/load more efficient by removing non-essential values
• Switch nav menu to icons
• Add convert DAG call
• Add hotkeys to menu functions
• Make input/output nodes more clear
• Check if file already exists on rename
• Prompt save if there are nodes on open
• Fix loss of state on refresh bug
• Put File details in File Browse
• Make Flask Backend URI configurable
• Add a last saved time to NavBar
• Add error handling to backend calls
• Only save local state if valid
• Add tests
• Get task descriptions from Operator list
• XSS and injection vulns?
• Ctl+Shift+F FIXME

Back-End Features

• Generate Operator Lists
• CLI to start Web and Front End
• Generate DAG Spec
• CLI to create new windmill project
• CLI to start windmill from a windmill project
• Implement windmill-dev start
• Save/Load Windmill Files functionality
• Get default values
• Pull parameters from parent classes
• Move airflow dependency as extra
• Convert WML into Python DAG
• API Endpoint to trigger WML -> DAG
• Make sure that nodes are being put in right order using ports
• Edge cases for WML -> DAG
• Allow repeated/weird dag/task ids (e.g. 123)
• Get WML owner and last-modified details during wml list
• Allow custom operators
• Strategy for Python Opjects (e.g. callables) - allow either a import ref or an inline statement
• Backport existing Python DAGs to WMLs
• Allow DAG updates to propogate to WMLs (probably better to just always backport - consolidating would be a mess)
• Add tests for different airflow version
• Version lock travis tox and poetry version?

Other features

• Validate on backend or front end or both?
• Doco
• Add permission restrictions for valid tags
• Only include dist folder from app in poetry build

Dev User Guide

To run as a dev:

• Clone from git
• Run poetry install -E airflow
• Run npm install from windmill/http/app
• Run windmill-dev start-frontend
• Run windmill-dev start-backend
• Open 127.0.0.1:1234

Run Python Tests

Expects Tox and Poetry to be available on path

pyenv install 3.6.5 3.7.7
pyenv local 3.6.5 3.7.7
tox

Future Usage Patterns

• Auto-sync for windmill project to git

Deployment

Deployment to PyPi is managed using Travis and should be done in the following steps:

• Run poetry version {patch|minor|major}
• Increment the version number in windmill/__init__.py
• Commit and merge code into the master branch
• Ensure that the travis build is green
• Create a git tag for the new build
• Push the tag to origin