stream-transform

What is stream-transform?

The stream-transform package is a simple and powerful tool for transforming data in Node.js streams. It allows you to easily modify data as it passes through a stream, making it highly useful for tasks such as data manipulation, filtering, and aggregation in real-time data processing applications.

What are stream-transform's main functionalities?

Synchronous Transformation

This feature allows for synchronous data transformation within a stream. The provided code sample demonstrates how to read data from a CSV file, transform each record by joining its elements with a space, and then output the transformed data.

const transform = require('stream-transform');
const transformer = transform(function(record, callback){
  setTimeout(function(){
    callback(null, record.join(' ') + '\n');
  }, 500);
}, {parallel: 5});
require('fs').createReadStream('/path/to/source.csv').pipe(transformer).pipe(process.stdout);

Asynchronous Transformation

This feature supports asynchronous transformation of stream data. The code sample shows how to asynchronously convert each piece of data to uppercase before passing it through the stream.

const transform = require('stream-transform');
const transformer = transform(function(record, callback){
  process.nextTick(function(){
    callback(null, record.toUpperCase());
  });
});
require('fs').createReadStream('/path/to/source.txt').pipe(transformer).pipe(process.stdout);

Other packages similar to stream-transform

through2

Through2 is a tiny wrapper around Node.js streams.Transform that makes it easier to create transform streams. It is similar to stream-transform in providing a simple API for data transformation but differs in its implementation and customization options.

event-stream

Event-stream is a toolkit for working with Node.js streams in a functional programming style, offering a variety of utilities for creating, combining, and consuming streams. While it provides more general stream manipulation capabilities, it shares the ability to transform stream data with stream-transform.

README

Stream transformation for Node.js and the web

The stream-transform project is a simple object transformation framework. It is part of the CSV project.

The Node.js stream.Transform API is implemented for scalability. The callback-based and sync APIs are also available for convenience. It is both easy to use and powerful.

Documentation

• Project homepage
• API
• Options
• Handler
• State properties
• Examples

Main features

• Extends the native Node.js transform stream API
• Simplicity with the optional callback and sync API
• Pipe transformations between readable and writable streams
• Synchronous versus asynchronous user functions
• Sequential and parallel execution
• Accept object, array or JSON as input and output
• Sequential or user-defined concurrent execution
• Skip and multiply records
• Alter or clone input records
• MIT License

Usage

Run npm install csv to install the full CSV module or run npm install csv-transform if you are only interested by the CSV stringifier.

The module is built on the Node.js Stream API. Use the callback and sync APIs for simplicity or the stream based API for scalability.

Example

The API is available in multiple flavors. This example illustrates the sync API.

import { transform } from 'stream-transform/sync';
import assert from 'assert';

const records = transform([
  [ 'a', 'b', 'c', 'd' ],
  [ '1', '2', '3', '4' ]
], function(record){
  record.push(record.shift());
  return record;
});

assert.deepEqual(records, [
  [ 'b', 'c', 'd', 'a' ],
  [ '2', '3', '4', '1' ]
]);

Development

Tests are executed with mocha. To install it, simple run npm install followed by npm test. It will install mocha and its dependencies in your project "node_modules" directory and run the test suite. The tests run against the CoffeeScript source files.

To generate the JavaScript files, run npm run coffee.

The test suite is run online with Travis. See the Travis definition file to view the tested Node.js version.

Contributors

The project is sponsored by Adaltas, an Big Data consulting firm based in Paris, France.

• David Worms: https://github.com/wdavidw