A library on top of either pex or conda-packto make your Python code easily available on a cluster
cluster-pack is a library on top of either pex or conda-pack to make your Python code easily available on a cluster.
Its goal is to make your prod/dev Python code & libraries easiliy available on any cluster. cluster-pack supports HDFS/S3 as a distributed storage.
The first examples use Skein (a simple library for deploying applications on Apache YARN) and PySpark with HDFS storage. We intend to add more examples for other applications (like Dask, Ray) and S3 storage.
An introducing blog post can be found here.
$ pip install cluster-pack
$ git clone https://github.com/criteo/cluster-pack
$ cd cluster-pack
$ pip install .
cluster-pack supports Python ≥3.7.
Ships a package with all the dependencies from your current virtual environment or your conda environment
Stores metadata for an environment
Supports "under development" mode by taking advantage of pip's editable installs mode, all editable requirements will be uploaded all the time, making local changes directly visible on the cluster
Interactive (Jupyter notebook) mode
Provides config helpers to directly use the uploaded zip file inside your application
Launching jobs from jobs by propagating all artifacts
FAQs
A library on top of either pex or conda-packto make your Python code easily available on a cluster
We found that cluster-pack demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 7 open source maintainers collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
Bybit's $1.46B hack by North Korea's Lazarus Group pushes 2025 crypto losses to $1.6B in just two months, already surpassing all of 2024's $1.49B total.
Security News
OpenSSF has published OSPS Baseline, an initiative designed to establish a minimum set of security-related best practices for open source software projects.
Security News
Michigan TypeScript founder Dimitri Mitropoulos implements WebAssembly runtime in TypeScript types, enabling Doom to run after processing 177 terabytes of type definitions.