Up-scale python functions for high performance computing (HPC) with executorlib.
Up-scale python functions for high performance computing (HPC) with executorlib.
The Python standard library provides the Executor interface with the ProcessPoolExecutor and the ThreadPoolExecutor for parallel execution of Python functions on a single computer. executorlib extends this functionality to distribute Python functions over multiple computers within a high performance computing (HPC) cluster. This can be either achieved by submitting each function as individual job to the HPC job scheduler with an HPC Cluster Executor - or by requesting a job from the HPC cluster and then distribute the Python functions within this job with an HPC Job Executor. Finally, to accelerate the development process executorlib also provides a Single Node Executor - to use the executorlib functionality on a laptop, workstation or single compute node for testing. Starting with the Single Node Executor:
from executorlib import SingleNodeExecutor
with SingleNodeExecutor() as exe:
future_lst = [exe.submit(sum, [i, i]) for i in range(1, 5)]
print([f.result() for f in future_lst])
In the same way executorlib can also execute Python functions which use additional computing resources, like multiple CPU cores, CPU threads or GPUs. For example if the Python function internally uses the Message Passing Interface (MPI) via the mpi4py Python libary:
from executorlib import SingleNodeExecutor
def calc(i):
from mpi4py import MPI
size = MPI.COMM_WORLD.Get_size()
rank = MPI.COMM_WORLD.Get_rank()
return i, size, rank
with SingleNodeExecutor() as exe:
fs = exe.submit(calc, 3, resource_dict={"cores": 2})
print(fs.result())
The additional resource_dict parameter defines the computing resources allocated to the execution of the submitted
Python function. In addition to the compute cores cores, the resource dictionary can also define the threads per core
as threads_per_core, the GPUs per core as gpus_per_core, the working directory with cwd, the option to use the
OpenMPI oversubscribe feature with openmpi_oversubscribe and finally for the Simple Linux Utility for Resource
Management (SLURM) queuing system the option to provide additional command line arguments
with the slurm_cmd_args parameter - resource dictionary
This flexibility to assign computing resources on a per-function-call basis simplifies the up-scaling of Python programs.
Only the part of the Python functions which benefit from parallel execution are implemented as MPI parallel Python
funtions, while the rest of the program remains serial.
The same function can be submitted to the SLURM job scheduler by replacing the
SingleNodeExecutor with the SlurmClusterExecutor. The rest of the example remains the same, which highlights how
executorlib accelerates the rapid prototyping and up-scaling of HPC Python programs.
from executorlib import SlurmClusterExecutor
def calc(i):
from mpi4py import MPI
size = MPI.COMM_WORLD.Get_size()
rank = MPI.COMM_WORLD.Get_rank()
return i, size, rank
with SlurmClusterExecutor() as exe:
fs = exe.submit(calc, 3, resource_dict={"cores": 2})
print(fs.result())
In this case the Python simple queuing system adapter (pysqa) is used to submit the
calc() function to the SLURM job scheduler and request an allocation with two CPU cores
for the execution of the function - HPC Cluster Executor. In the background the sbatch
command is used to request the allocation to execute the Python function.
Within a given SLURM job executorlib can also be used to assign a subset of the available computing resources to execute a given Python function. In terms of the SLURM commands, this functionality internally uses the srun command to receive a subset of the resources of a given queuing system allocation.
from executorlib import SlurmJobExecutor
def calc(i):
from mpi4py import MPI
size = MPI.COMM_WORLD.Get_size()
rank = MPI.COMM_WORLD.Get_rank()
return i, size, rank
with SlurmJobExecutor() as exe:
fs = exe.submit(calc, 3, resource_dict={"cores": 2})
print(fs.result())
In addition, to support for SLURM executorlib also provides support for the hierarchical flux job scheduler. The flux job scheduler is developed at Larwence Livermore National Laboratory to address the needs for the up-coming generation of Exascale computers. Still even on traditional HPC clusters the hierarchical approach of the flux is beneficial to distribute hundreds of tasks within a given allocation. Even when SLURM is used as primary job scheduler of your HPC, it is recommended to use SLURM with flux as hierarchical job scheduler within the allocations.
FAQs
Up-scale python functions for high performance computing (HPC) with executorlib.
We found that executorlib demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 2 open source maintainers collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
npm now supports Trusted Publishing with OIDC, enabling secure package publishing directly from CI/CD workflows without relying on long-lived tokens.
Research
/Security News
A RubyGems malware campaign used 60 malicious packages posing as automation tools to steal credentials from social media and marketing tool users.
Security News
The CNA Scorecard ranks CVE issuers by data completeness, revealing major gaps in patch info and software identifiers across thousands of vulnerabilities.