parallel-transform

What is parallel-transform?

The parallel-transform npm package is a Node.js module that allows you to perform parallel transformations on streams of data. It is particularly useful when you need to process data in a stream without overloading the system with too many concurrent operations. It maintains the order of the stream and allows you to specify the maximum number of concurrent transformations.

What are parallel-transform's main functionalities?

Parallel Stream Transformation

This feature allows you to create a transform stream that can process data in parallel. The code sample shows how to create a parallel transform stream that converts input data to uppercase with a maximum of 10 concurrent operations.

const ParallelTransform = require('parallel-transform');
const stream = require('stream');

const parallelStream = new ParallelTransform(10, function(data, callback) {
  // Perform some asynchronous operation
  setTimeout(() => {
    callback(null, data.toString().toUpperCase());
  }, 100);
});

process.stdin.pipe(parallelStream).pipe(process.stdout);

Other packages similar to parallel-transform

through2-concurrent

through2-concurrent is similar to parallel-transform in that it allows you to process streams of data concurrently. It provides a simple API to create a transform stream with concurrency control. The main difference is that through2-concurrent is built on top of through2, which is a thin wrapper around Node.js streams.Transform.

async-transform

async-transform is another package that offers similar functionality to parallel-transform. It allows for asynchronous transformation of stream data with concurrency control. However, it differs in its API and the way it handles the transformation function, which is based on the async module.

stream-each

stream-each provides a way to iterate over the data in a stream and apply an asynchronous function to each chunk of data. While it does not directly create a transform stream, it can be used in combination with other stream methods to achieve parallel processing. It is more focused on iteration rather than transformation.

README

parallel-transform

Transform stream for Node.js that allows you to run your transforms in parallel without changing the order of the output.

npm install parallel-transform

It is easy to use

var transform = require('parallel-transform');

var stream = transform(10, function(data, callback) { // 10 is the parallism level
	setTimeout(function() {
		callback(null, data);
	}, Math.random() * 1000);
});

for (var i = 0; i < 10; i++) {
	stream.write(''+i);
}
stream.end();

stream.on('data', function(data) {
	console.log(data); // prints 0,1,2,...
});
stream.on('end', function() {
	console.log('stream has ended');
});

If you run the above example you'll notice that it runs in parallel (does not take ~1 second between each print) and that the order is preserved

Stream options

All transforms are Node 0.10 streams. Per default they are created with the options {objectMode:true}. If you want to use your own stream options pass them as the second parameter

var stream = transform(10, {objectMode:false}, function(data, callback) {
	// data is now a buffer
	callback(null, data);
});

fs.createReadStream('filename').pipe(stream).pipe(process.stdout);

Unordered

Passing the option {ordered:false} will output the data as soon as it's processed by a transform, without waiting to respect the order.

License

MIT