Script of Scripts (SoS): an interactive, cross-platform, and cross-language workflow system for reproducible data analysis
Script of Scripts (SoS) consists of SoS Notebook, a Jupyter-based polyglot notebook that allows the use of multiple Jupyter kernels in one notebook, and
SoS Workflow, a workflow system for the execution of workflows in both process- and outcome-oriented styles. It is designed for data scientists and bioinformatics who routinely work with scripts in different languages such as bash, Python, R, and SAS. This repository contains the SoS Workflow workflow engine.
Please refer to relevant publications SoS Notebook: An Interactive Multi-Language Data Analysis Environment in Bioinformatics, Script of Scripts: a pragmatic workflow system for daily computational research in PLoS Computational Biology, and the SoS homepage for more information on SoS Notebook and SoS Workflow.
We welcome and value community contributions: please post issues to provide us feedback or get our support; please send pull requests if you have helped fixing bugs or making improvements to the source code. In addition, if you are interesting in extending SoS, for example adding a new target type or task engine, please have a look at our tutorial on extending SoS here.
If you are using conda, you can install sos using command
conda install sos -c conda-forge
You can also install a majority of the SoS suite of tools using commands such as
conda install sos sos-pbs sos-notebook jupyterlab-sos sos-bash sos-python sos-r -c conda-forge
This will install SoS Workflow, SoS Notebook and its JupyterLab extension, language modules for
Bash, Python2, Python3, R, R itself (r-base and r-irkernel etc) and needed
libraries (e.g. feather) if needed.
If you are not using conda, you will have to install all the pieces one by one, but you will also have the freedom to use non-conda installation of R, Julia etc. With a working Python 3.6 installation, you can install SoS Workflow with command
pip install sos
and also
pip install sos-pbs
if you would like to use SoS with a remote batch system such as LSF or Slurm.
You can install SoS Notebook, and register the sos kernel with Jupyter using the following commands
pip install sos-notebook
python -m sos_notebook.install
To exchange data between live kernels in SoS Notebook, you will need to install individual kernels (e.g. irkernel for R), make sure they work under Jupyter, and install SoS language modules using commands such as
pip install sos-r sos-matlab sos-python sos-bash
Different modules might be needed to assist data exchange among kernels. Please refer to the installation page of sos website for details.
SoS 0.25.0:
remote target type, remote push and pull, named (#name) path because SoS now assumes that all input and output files are available on shared drives with identical path names.SoS 0.24.5:
SoS 0.22.5:
SoS Notebook 0.22.3
SoS Notebook 0.22.2
SoS 0.21.12
pem_file to authenticate to remote hosts.SoS 0.20.9
-r host to allow workflow to be executed with templates and on PBS-b BIN_DIR due to sos#1319SoS 0.19.15
trunk_workers.-j of commands sos run and sos execute to use remote workers.-q from localhost to None (no task).SoS 0.19.0
-j.SoS Notebook 0.18.5
SoS 0.18.6
output_from(step) to obtain output from a workflow.SoS 0.18.5
-T to enfore dependency tracing.traced to make dependencies that will always be tracedSoS 0.18.4
depends statement always try to build dependency.SoS Notebook 0.18.1
& to the end of magics %run, %sosrun, and %runfile to execute workflow in background.%rerun and add magic %runfile-r to %save to execute the cell after saving it.SoS 0.18.0
output_from and named_output to support named inputs and outputs, among other new features summarized in this ticket.group_by, paired_with etc for functions output_from and named_output.concurrent=True as default for substep execution.stop_if and replace it with done_if and skip_ifsos_variable to import shared variable in a stepSoS Notebook 0.18.0
SoS 0.9.16.10
SoS 0.9.16.0, SoS Notebook 0.9.16.0
SoS 0.9.15.1
SoS 0.9.14.10
sos_step() target that matches multiple steps.SoS 0.9.14.3:
-p to generate a summary report after the completion of workflow.sos run script -h).SoS Notebook 0.9.14.4:
SoS Notebook 0.9.14.1
%revisions magic to display revision history of the current document.SoS 0.9.14.1
SoS Notebook 0.9.13.4
%cd now changes directory of all subkernelsSoS Notebook 0.9.12.12
SoS 0.9.12.11
zap() to SoS path classes path, paths, file_target, and sos_targetsSoS Notebook 0.9.12.11
%clear --classSoS 0.9.12.9
active of actions and tasks to accept conditions.~) for SoS path types path, paths and file_targets.SoS Notebook 0.9.12.9
SoS 0.9.12.3
SoS 0.9.11.3
stdout and stderr to redict output from script-executing actions.--append to magic %capture .SoS 0.9.11.2
python -m sos_notebook.install to install sos kernel to Jupyter.SoS 0.9.10.19
concurrent=True to allow parallel execution of input groups.SoS Notebook 0.9.10.17
%capture to capture output of cellsFAQs
Script of Scripts (SoS): an interactive, cross-platform, and cross-language workflow system for reproducible data analysis
We found that sos demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 2 open source maintainers collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
Research
The Socket Research Team breaks down a malicious wrapper package that uses obfuscation to harvest credentials and exfiltrate sensitive data.
Research
Security News
Attackers used a malicious npm package typosquatting a popular ESLint plugin to steal sensitive data, execute commands, and exploit developer systems.
Security News
The Ultralytics' PyPI Package was compromised four times in one weekend through GitHub Actions cache poisoning and failure to rotate previously compromised API tokens.