The parallel-transform npm package is a Node.js module that allows you to perform parallel transformations on streams of data. It is particularly useful when you need to process data in a stream without overloading the system with too many concurrent operations. It maintains the order of the stream and allows you to specify the maximum number of concurrent transformations.

What are parallel-transform's main functionalities?

Parallel Stream Transformation

This feature allows you to create a transform stream that can process data in parallel. The code sample shows how to create a parallel transform stream that converts input data to uppercase with a maximum of 10 concurrent operations.

const ParallelTransform = require('parallel-transform');
const stream = require('stream');

const parallelStream = new ParallelTransform(10, function(data, callback) {
  // Perform some asynchronous operation
  setTimeout(() => {
    callback(null, data.toString().toUpperCase());
  }, 100);
});

process.stdin.pipe(parallelStream).pipe(process.stdout);

Other packages similar to parallel-transform

Readme

Source

parallel-transform

Transform stream for Node.js that allows you to run your transforms in parallel without changing the order of the output.

npm install parallel-transform

It is easy to use

var transform = require('parallel-transform');

var stream = transform(10, function(data, callback) { // 10 is the parallism level
	setTimeout(function() {
		callback(null, data);
	}, Math.random() * 1000);
});

for (var i = 0; i < 10; i++) {
	stream.write(''+i);
}
stream.end();

stream.on('data', function(data) {
	console.log(data); // prints 0,1,2,...
});
stream.on('end', function() {
	console.log('stream has ended');
});

If you run the above example you'll notice that it runs in parallel (does not take ~1 second between each print) and that the order is preserved

Stream options

All transforms are Node 0.10 streams. Per default they are created with the options {objectMode:true}. If you want to use your own stream options pass them as the second parameter

var stream = transform(10, {objectMode:false}, function(data, callback) {
	// data is now a buffer
	callback(null, data);
});

fs.createReadStream('filename').pipe(stream).pipe(process.stdout);

Unordered

Passing the option {ordered:false} will output the data as soon as it's processed by a transform, without waiting to respect the order.

License

MIT

Keywords

FAQs

What is parallel-transform?

Is parallel-transform popular?

Is parallel-transform well maintained?

Last updated on 05 Sep 2019

Did you know?

Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.

Install