TensorHive is an open source tool for managing computing resources used by multiple users across distributed hosts. It focuses on granting exclusive access to GPUs for machine learning workloads and consists of reservation, monitoring and job execution modules.
It's designed with simplicity, flexibility and configuration-friendliness in mind.
Each column represents all reservation events for a GPU on a given day.
In order to make a new reservation simply click and drag with your mouse, select GPU(s), add some meaningful title, optionally adjust time range.
If there are many hosts and GPUs in our infrastructure, you can use our simplified, horizontal calendar to quickly identify empty time slots and filter out already reserved GPUs.
From now on, only your processes are eligible to run on the reserved GPU(s). TensorHive periodically checks if some other user has violated it. They will be spammed with warnings on all his PTYs, emailed every once in a while, additionally admin will also be notified (it all depends on the configuration).
| Terminal warning | Email warning | Admin warning |
|---|---|---|
 |  |  |
Accessible infrastructure can be monitored in the Nodes overview tab. Sample screenshot: Here you can add new watches, select metrics and monitor ongoing GPU processes and their owners.
Thanks to the Job execution module, you can define commands for tasks you want to run on any configured nodes.
You can manage them manually, set specific spawn/terminate dates or add jobs to a queue, so that they are executed automatically
when the required resources are not reserved.
Commands are run within screen session, so attaching to it while they are running is a piece of cake.
It provides a simple, but flexible (framework-agnostic) command templating mechanism that will help you automate multi-node trainings.
Additionally, specialized templates help to conveniently set proper parameters for chosen well known frameworks. In the examples directory, you will find sample scenarios of using the Job execution module for various
frameworks (including TensorFlow and PyTorch) and computing environments.
TensorHive requires that users who want to use this feature must append TensorHive's public key to their ~/.ssh/authorized_keys on all nodes they want to connect to.
Our goal is to provide solutions for painful problems that ML engineers often have to struggle with when working with remote machines in order to run neural network trainings.
gpu_util, mem_util, mem_used are great for this purpose:zero: Dead-simple one-machine installation and configuration, no sudo requirements
:one: Users can make GPU reservations for specific time range in advance via reservation mechanism
:arrow_right: no more frustration caused by rules: "first come, first served" or "the law of the jungle".
:two: Users can prepare and schedule custom tasks (commands) to be run on selected GPUs and hosts
:arrow_right: automate and simplify distributed trainings - "one button to rule them all"
:three: Gather all useful GPU metrics, from all configured hosts in one dashboard
:arrow_right: no more manual logging in to each individual machine in order to check if GPU is currently in use or not
:four: Access to specific GPUs or hosts can be granted to specific users or groups
:arrow_right: division of the infrastructure can be easily adjusted to the current needs of work groups in your organization
:five: Automatic execution of queued jobs when there are no active GPU reservations
:arrow_right: jobs that are not urgent can be added to a queue and automatically executed later
nvidia-smi command)tensorhive) and adding this user to the tty system group.pip install tensorhive
(optional) For development purposes we encourage separation from your current python packages using e.g. virtualenv, Anaconda.
git clone https://github.com/roscisz/TensorHive.git && cd TensorHive
pip install -e .
TensorHive is already shipped with newest web app build, but in case you modify the source, you can can build it with make app. For more useful commands see our Makefile.
Build tested with Node v14.15.4 and npm 6.14.10
The init command will guide you through basic configuration process:
tensorhive init
You can check connectivity with the configured hosts using the test command.
tensorhive test
(optional) If you want to allow your UNIX users to set up their TensorHive accounts on their own and run distributed
programs through Job execution module, use the key command to generate the SSH key for TensorHive:
tensorhive key
Now you should be ready to launch a TensorHive instance:
tensorhive
Web application and API Documentation can be accessed via URLs highlighted in green (Ctrl + click to open in browser).
You can fully customize TensorHive behaviours via INI configuration files (which will be created automatically after tensorhive init):
~/.config/TensorHive/hosts_config.ini
~/.config/TensorHive/main_config.ini
~/.config/TensorHive/mailbot_config.ini
Serving TensorHive through reverse proxy requires proper configuration of URL parameters in the [api] section of
main_config.ini, including url_schema, url_hostname, url_port and url_prefix.
Let's assume that the WebApp is served locally on http://localhost:5000, the API on http://localhost:1111 and we
want to serve TensorHive publicly at https://some-server/tensorhive. In such case the following main_config.ini:
url_schema = https
url_hostname = some-server
url_port = 443
url_prefix = tensorhive/api
should be used along with a reverse proxy similar to the following example for nginx:
location /tensorhive {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-Host $host:$server_port;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
add_header 'Access-Control-Allow-Origin' '*';
add_header 'Access-Control-Allow-Credentials' 'true';
add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
add_header 'Access-Control-Allow-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type';
proxy_pass http://localhost:5000/tensorhive;
proxy_set_header SCRIPT_NAME /tensorhive;
}
location /tensorhive/api {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-Host $host:$server_port;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
add_header 'Access-Control-Allow-Origin' '*';
add_header 'Access-Control-Allow-Credentials' 'true';
add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
add_header 'Access-Control-Allow-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type';
proxy_pass http://localhost:1111;
}
If you use TensorHive for a scientific publication, we would appreciate citations:
@article{JMLR:v22:20-225,
author = {Paweł Rościszewski and Michał Martyniak and Filip Schodowski},
title = {TensorHive: Management of Exclusive GPU Access for Distributed Machine Learning Workloads},
journal = {Journal of Machine Learning Research},
year = {2021},
volume = {22},
number = {215},
pages = {1-5},
url = {http://jmlr.org/papers/v22/20-225.html}
}
We'd :heart: to collect your observations, issues and pull requests!
Feel free to report any configuration problems, we will help you.
Currently we are gathering practical infrastructure protection scenarios from our users to extract and further support the most common TensorHive deployments.
If you consider becoming a contributor, please look at issues labeled as good-first-issue and help wanted.
Project created and maintained by:
Michał Martyniak (@micmarty)Top contributors:
TensorHive has been greatly supported within a joint project between VoiceLab.ai and Gdańsk University of Technology titled: "Exploration and selection of methods for parallelization of neural network training using multiple GPUs".
FAQs
A user-friendly GPU management tool for distributed machine learning workloads
We found that tensorhive demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
In this segment of the Risky Business podcast, Feross Aboukhadijeh and Patrick Gray discuss the challenges of tracking malware discovered in open source softare.
Research
Security News
A threat actor's playbook for exploiting the npm ecosystem was exposed on the dark web, detailing how to build a blockchain-powered botnet.
Security News
NVD’s backlog surpasses 20,000 CVEs as analysis slows and NIST announces new system updates to address ongoing delays.