progress-parallel
Readme
- Some serious bugs were fixed. -
Serious bugs are fixed as hotfixes in v3.1.1.
We have discovered a serious flaw in the functionality available in for-loop added in v3.0.1.
This has been corrected in v3.1.1.
Accordingly, v3.0.1 has been removed from PyPI.
prpl(progress-parallel) is a library to visualize the progress of parallel processing by concurrent.futures, the standard python library.
prpl is a "Tips" library that makes the standard python parallel processing library simpler to use.
The general functionality is the same as concurrent.futures itself, but it is possible to visualize the parallel processing status of threads generated using this library.
In prpl, when performing calculations on numbers managed by a list, the list is automatically parallelized and parallelized.
The progress of the parallel processing can be visualized and the parallel processing can be checked.
Standard features include the following,
Display of thread progress
res = prpl(target_list=t_list, target_function=test_calc_func)
Display of progress bar
res = prpl(target_list=t_list, target_function=test_calc_func, symbol="#", smpl=True)
Measure run timer
res = prpl(target_list=t_list, target_function=test_calc_func, timer=True)
Change the color of the progress bar.
res = prpl(target_list=t_list, target_function=test_calc_func, symbol="#", color="green")
Only single progress bar is available.
res = prpl(target_list=t_list, target_function=test_calc_func, list_sep=1)
For use with the for-loop.
for _ in prpl(t_list, symbol="#"):
pass
When used in a for-loop, the color of the symbol can also be changed.However, this feature does not allow for graphical functions with arrows, such as multi-threading (parallel processing) mode. If the symbol argument is not used, a standard count-up progress bar is used.
for _ in prpl(t_list):
pass
When multiple arguments are passed to a function.
""" For functions with multiple arguments. """
def multi_arguments_target_function(a,b,c,d):
# The argument that receives the elements of the list is "a".
return a+b+c+d
args_dict = {
"b": 2,
"c": 3,
"d": 4
}
res = prpl(target_list=t_list, target_function=multi_arguments_target_function, args=args_dict)
When passing multiple arguments to a function, be sure to pass them as type of dict.
Also, it is not necessary to include the target list in the arguments grouped together in that type of dict.
For other samples, test code is available at tests directory. Please run it.
(and gifs are at sample git directory)
For parallel processing, it is CPU parallel processing using the standard python library.
For more information, check the python concurrent.futures documentation.
An actual code sample example would be as follows. An example of implementation in actual code is shown below.
When performing some calculation process on a set of numbers compiled in a list, the conventional implementation is as follows:
""" conventional method """
calculation_result = []
target_list = list(range(0, 100000))
for target_var in target_list:
calc_answer = (target_var**target_var)**2 # formula
calculation_result.append(calc_answer)
By using prpl, the process inside a for loop statement is automatically executed in parallel by passing it as a function, and the result is returned.
""" Separate formulas to be processed in the for loop as functions """
def target_function(target_var):
calc_answer = (target_var**target_var)**2 # target formula
return calc_answer
""" Methods using prpl """
from prpl import prpl
target_list = list(range(0, 100000))
calculation_result = prpl(target_list, target_function) # prpl
The actual operating gif of the comparison is as follows:
It is important to note that if the formula is not complex, it is not suitable for parallel processing. Because of the nature of parallel processing, prpl should not be used unnecessarily.
The list of arguments, etc. that can be used in prpl is as follows.
result_list = prpl( # The results will always return with a type of List.
target_list, # Required argument
### List to be used in the target function.
target_function, # May be required argument.
### A function that contains a target formula, etc, and must include a return statement.
### Required arguments if prpl is used in parallel processing visualization.
### It is not required if it is used in the for-loop.
args,
### Multiple arguments managed by dictionary type.
list_sep,
### Number of list divisions.
### Default setting is 5.
checkpoint,
### Number of breaks to display progress.
### ### Default setting is 10.
title,
### Title of each progress bar.
### By default, the name of the target function.
symbol,
### Progress bar mark settings.
symbol_c,
### Setting the length of the progress bar.
### Default setting is 50.
smpl,
### Simple display mode.
### Default setting is False.
timer,
### Measuring run time.
### Not available in simple mode.
color
### Change the color of the progress bar.
)
The enumerate() is not available.
Normally, enumerate() is available in a for loop statement, but this is not available in a ptpl using concurrent.futures.
A print() cannot be used within a function for which parallel processing is desired.
Strictly speaking, you can use it. However, using a print() is the same as executing a print statement inside a for loop, so the progress bar will be misaligned.
Since parallel processing conforms to the concurrent.futures library, anything that cannot be done with concurrent.futures is not possible with prpl.
Concurrent.futures is used in prpl parallel processing. Therefore, what is impossible with concurrent.futures is also impossible with prpl. To the last, prpl is a library to handle parallel processing in a simple and intuitive way, and a library to visualize the parallel processing.
The display may shift on the console.
During execution, the vertical display of the console is controlled in the program, but the horizontal length is not. Therefore, if the prpl display exceeds the length of the console's width, the prpl may be significantly disturbed.
Be sure to provide a target function with return.
By the nature of the program, it performs an arithmetic operation on the given "list-type" array and returns the result in a list type. Therefore, prpl will cause an error if some return is not made in the function.
The following are appropriate and inappropriate examples.
""" appropriate example target function """
def OK_func(i):
ans = (i**i)**2
return ans # return
""" inappropriate example """
def NG_func(i):
ans = (i**i)**2
# not return
Be sure to include a return statement in the function.
pip install progress-parallel
pip install git+https://github.com/Domzou-kun/prpl.git
or install via SSH
pip install git+git://github.com:Domzou-kun/prpl.git
prpl's, the progressbar on the console uses ESC and CSI sequences to control multiple lines simultaneously.
When using CSI and ESC sequences with cmd, etc., terminal settings must be enabled.
However, many of the programs that display such a progressbar as this one have modified kernel and console settings to suit their individual programs. This means that if other console-controlling programs are used at the same time, the settings may conflict and break the drawing.
We have already confirmed that when tqdm and prpl are imported and used at the same time, there is a conflict with tqdm's settings and prpl's display is corrupted.
We are currently working on a fix for this problem.
If you have a suggestion for a technical solution, please write to the issue. We will gladly review and consider it.
Domzou
The link to PyPI is here.
このreadmeを日本語にしたものになります。本readmeの重要な部分をメインに日本語に直してQiitaに投稿しています。
If you want to know about past versions, please refer to version history.
PyTorch has a MIT license, as found in the LICENSE file.
FAQs
multi threading progress bar
We found that progress-parallel demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
The Python Software Foundation has secured a 5-year sponsorship from Fastly that supports PSF's activities and events, most notably the security and reliability of the Python Package Index (PyPI).
Security News
LDAPjs, an LDAP Client and Server API for Node.js, was decommissioned after its maintainer received an abusive email from a user, raising concerns about this form of abuse as a potential attack vector.
Security News
CISA launched a new project called Vulnrichment to enrich CVEs with details that help prioritize patching and mitigation efforts, as the NVD backlog of unenriched CVEs awaiting analysis surpasses 10,000.