parallelQueue

= JobQueue - Let run your jobs in parallel

This package contains jobQueue:

• A simple class to parallalize your work on a user definend number of threads
• A ruby script, which uses the above class to run each line of a shell script

jobQueue can do the following things:

• Run blocks, Procs and Lambdas
• Run instance and class methods
• Run shell commands
• Respect user definded locks

I started a python2 implementation of this, which can be installed via pip (https://pypi.python.org/pypi/jobqueue). When its maature enough, I will use the ruby version numbers for it.

== Installation

=== Gem Installation

Download and install jobQueue with the following.

gem install jobQueue

=== Requirements

JobQueue requires Ruby only, but versions 1.9.x are needed to make use of system threads.

== Usage

=== Parallelize Ruby's blocks, procs, lambdas and things

Create a JobQueue with nThreads worker with:

jq = JobQueue.new(nThreads)

Use its push method to put in something to do

• For blocks: jq.push do myObject.method0(...) myObject.method1(...) myObject.method3(...) end

• For procs and lambdas:

  jp.push(myProc,arg0,arg1,...)

• For object methods:

  jq.push([myObject,[:method,arg0,arg1,...])

• Same code can be used for class methods:

  jq.push(myClass,[:myClassMethod,arg0,arg1,...])

To start the workers, call

jq.run

That's it. You might have look at tests, that come with the jobQueue gem.

=== Parallelize system commands

Use a separate class for this and push string to it:

jq = SystemJobs.new(nThreads) jq.push('find ./src -name ".rb"','find ./downloads -name ".rb"')

will start 2 parallel searches for ruby files. This is implemented in the prun.rb script, which comes with the jobQueue gem. To run each line of a shell script on 10 threads, use:

prun.rb -j 10 jobs.sh

The '-j' switch is optional. Default is the maximum number of cores. The script accepts multiple files and processes one after another. Try '-h' for documentation.

== Support, Issues, Bugs, ...

please use personal mail, ruby-lang mailing list or github

== Changelog

• 1.0.11: prun.rb now ignores empty lines
• 1.0.10: more flexible logging control (new switches '-l' and '-b')
• 1.0.9: print out stdout and stderr from the jobs given to prun.rb, use '-D' to avoid printing
• 1.0.8: support AIX for getting the maximum number of processors, improve processor count for jruby and rbx

== Credits

[Robert Klemme] For the first hints: https://www.ruby-forum.com/topic/68001#86298

== License

jobQueue use the BSD License

:include LICENSE

= Other stuff

Author:: Ralf Mueller [e-mail] Requires:: Ruby 1.9 or later License:: Copyright 2011-2016 by Ralf Mueller Released under BSD-style license. See the LICENSE file included in the distribution.