batch-cluster

batch-cluster

Support external batch-mode tools within Node.js.

Many command line tools, like ExifTool, PowerShell, and GraphicsMagick, support running in a "batch mode" that accept a series of discrete commands provided through stdin and results through stdout. As these tools can be fairly large, spinning them up can be expensive (especially on Windows).

This module expedites these commands, or "Tasks," by managing a cluster of these "batch" processes and queue of pending tasks, feeding processes pending tasks when they are idle. Tasks are monitored for errors and have timeouts, which cause the host process to be recycled. Batch processes are also recycled after processing N tasks or running for N seconds, in an effort to minimize the impact of memory leaks.

As of version 4, retry logic for tasks is a separate concern from this module.

This package powers exiftool-vendored, whose source you can examine as an example consumer.

Installation

Depending on your yarn/npm preference:

$ yarn add batch-cluster
# or
$ npm install --save batch-cluster

Changelog

See CHANGELOG.md.

Usage

The child process must use stdin and stdout for control/response. BatchCluster will ensure a given process is only given one task at a time.

1. Create a singleton instance of BatchCluster.

   Note the constructor options takes a union type of

   • ChildProcessFactory and
   • BatchProcessOptions, both of which have no defaults, and
   • BatchClusterOptions, which has defaults that may or may not be relevant to your application.

2. The default logger writes warning and error messages to console.warn and console.error. You can change this to your logger by using setLogger.

3. Implement the Parser class to parse results from your child process.

4. Construct a Task with the desired command and the parser you built in the previous step, and submit it to your BatchCluster singleton's enqueueTask method.

See src/test.ts for an example child process. Note that the script is designed to be flaky on order to test BatchCluster's retry and error handling code.

Changelog

v5.6.8

• 📦 Updated development dependencies (new TypeScript)