jest-worker
Module for executing heavy tasks under forked processes in parallel, by
providing a Promise
based interface, minimum overhead, and bound workers.
The module works by providing an absolute path of the module to be loaded in all
forked processes. Files relative to a node module are also accepted. All methods
are exposed on the parent process as promises, so they can be await
'ed. Child
(worker) methods can either be synchronous or asynchronous.
The module also implements support for bound workers. Binding a worker means
that, based on certain parameters, the same task will always be executed by the
same worker. The way bound workers work is by using the returned string of the
computeWorkerKey
method. If the string was used before for a task, the call
will be queued to the related worker that processed the task earlier; if not, it
will be executed by the first available worker, then sticked to the worker that
executed it; so the next time it will be processed by the same worker. If you
have no preference on the worker executing the task, but you have defined a
computeWorkerKey
method because you want some of the tasks to be sticked,
you can return null
from it.
The list of exposed methods can be explicitly provided via the exposedMethods
option. If it is not provided, it will be obtained by requiring the child module
into the main process, and analyzed via reflection. Check the "minimal example"
section for a valid one.
Install
$ yarn add jest-worker
Example
This example covers the minimal usage:
File parent.js
import Worker from 'jest-worker';
async function main() {
const worker = new Worker(require.resolve('./worker'));
const result = await worker.hello('Alice');
}
main();
File worker.js
export function hello(param) {
return 'Hello, ' + param;
}
API
The only exposed method is a constructor (Worker
) that is initialized by
passing the worker path, plus an options object.
workerPath: string
(required)
Node module name or absolute path of the file to be loaded in the child
processes. Use require.resolve
to transform a relative path into an absolute
one.
options: Object
(optional)
exposedMethods: $ReadOnlyArray<string>
(optional)
List of method names that can be called on the child processes from the parent
process. You cannot expose any method named like a public Worker
method, or
starting with _
. If you use method auto-discovery, then these methods will not
be exposed, even if they exist.
numWorkers: number
(optional)
Amount of workers to spwan. Defaults to the number of CPUs minus 1.
maxRetries: number
(optional)
Maximum amount of times that a dead child can be re-spawned, per call. Defaults
to 3
, pass Infinity
to allow endless retries.
forkOptions: Object
(optional)
Allow customizing all options passed to childProcess.fork
. By default, some
values are set (cwd
and env
), but you can override them and customize the
rest. For a list of valid values, check
the Node documentation.
computeWorkerKey: (method: string, ...args: Array<any>) => ?string
(optional)
Every time a method exposed via the API is called, computeWorkerKey
is also
called in order to bound the call to a worker. This is useful for workers that
are able to cache the result or part of it. You bound calls to a worker by
making computeWorkerKey
return the same identifier for all different calls. If
you do not want to bind the call to any worker, return null
.
The callback you provide is called with the method name, plus all the rest of
the arguments of the call. Thus, you have full control to decide what to return.
Check a practical example on bound workers under the "bound worker usage"
section.
By default, no process is bound to any worker.
Worker
The returned Worker
instance has all the exposed methods, plus some additional
ones to interact with the workers itself:
getStdout(): Readable
Returns a ReadableStream
where the standard output of all workers is piped.
Note that the silent
option of the child workers must be set to true
to make
it work. This is the default set by jest-worker
, but keep it in mind when
overriding options through forkOptions
.
getStderr(): Readable
Returns a ReadableStream
where the standard error of all workers is piped.
Note that the silent
option of the child workers must be set to true
to make
it work. This is the default set by jest-worker
, but keep it in mind when
overriding options through forkOptions
.
end()
Finishes the workers by killing all workers. No further calls can be done to the
Worker
instance.
More examples
Standard usage
This example covers the standard usage:
File parent.js
import Worker from 'jest-worker';
async function main() {
const myWorker = new Worker(require.resolve('./worker'), {
exposedMethods: ['foo', 'bar'],
numWorkers: 4,
});
console.log(await myWorker.foo('Alice'));
console.log(await myWorker.bar('Bob'));
myWorker.end();
}
main();
File worker.js
export function foo(param) {
return 'Hello from foo: ' + param;
}
export function bar(param) {
return 'Hello from bar: ' + param;
}
Bound worker usage:
This example covers the usage with a computeWorkerKey
method:
File parent.js
import Worker from 'jest-worker';
async function main() {
const myWorker = new Worker(require.resolve('./worker'), {
computeWorkerKey: (method, filename) => filename,
});
console.log(await myWorker.transform('/tmp/foo.js'));
await sleep(10000);
console.log(await myWorker.transform('/tmp/foo.js'));
myWorker.end();
}
main();
File worker.js
import babel from 'babel-core';
const cache = Object.create(null);
export function transform(filename) {
if (cache[filename]) {
return cache[filename];
}
return new Promise((resolve, reject) => {
babel.transformFile(filename, (err, result) => {
if (err) {
reject(err);
} else {
resolve((cache[filename] = result));
}
});
});
}