labeled-stream-splicer

What is labeled-stream-splicer?

The labeled-stream-splicer package allows you to create a pipeline of streams that can be dynamically modified by adding, removing, or replacing streams at specific labels. This is particularly useful for complex stream processing tasks where the pipeline needs to be adjusted on the fly.

What are labeled-stream-splicer's main functionalities?

Creating a basic pipeline

This code demonstrates how to create a basic pipeline with two labeled steps. Each step is a through stream that logs the chunk it processes.

const splicer = require('labeled-stream-splicer');
const through = require('through2');

const pipeline = splicer([
  'step1', through.obj((chunk, enc, callback) => {
    console.log('Step 1:', chunk.toString());
    callback(null, chunk);
  }),
  'step2', through.obj((chunk, enc, callback) => {
    console.log('Step 2:', chunk.toString());
    callback(null, chunk);
  })
]);

pipeline.write('Hello, World!');
pipeline.end();

Dynamically adding a stream

This code shows how to dynamically add a new stream to the pipeline. The new stream is inserted before the existing 'step1' stream.

const splicer = require('labeled-stream-splicer');
const through = require('through2');

const pipeline = splicer([ 'step1', through.obj((chunk, enc, callback) => {
  console.log('Step 1:', chunk.toString());
  callback(null, chunk);
})]);

pipeline.write('Hello, World!');
pipeline.end();

pipeline.splice('step1', 0, through.obj((chunk, enc, callback) => {
  console.log('New Step:', chunk.toString());
  callback(null, chunk);
}));

pipeline.write('Hello again!');
pipeline.end();

Removing a stream

This code demonstrates how to remove a stream from the pipeline. The 'step2' stream is removed, so only 'step1' processes the input.

const splicer = require('labeled-stream-splicer');
const through = require('through2');

const pipeline = splicer([ 'step1', through.obj((chunk, enc, callback) => {
  console.log('Step 1:', chunk.toString());
  callback(null, chunk);
}), 'step2', through.obj((chunk, enc, callback) => {
  console.log('Step 2:', chunk.toString());
  callback(null, chunk);
})]);

pipeline.splice('step2', 1);

pipeline.write('Hello, World!');
pipeline.end();

Other packages similar to labeled-stream-splicer

stream-combiner2

stream-combiner2 allows you to combine multiple streams into one. Unlike labeled-stream-splicer, it does not support dynamic modification of the pipeline but is simpler to use for static pipelines.

multipipe

multipipe is another package for combining multiple streams into a single pipeline. It is similar to stream-combiner2 but provides better error handling. It also lacks the dynamic modification capabilities of labeled-stream-splicer.

pumpify

pumpify combines an array of streams into a single duplex stream. It is designed to handle backpressure correctly but does not support dynamic pipeline modifications like labeled-stream-splicer.

README

labeled-stream-splicer

stream splicer with labels

example

Here's an example that exposes a label for deps and pack:

var splicer = require('labeled-stream-splicer');
var through = require('through2');
var deps = require('module-deps');
var pack = require('browser-pack');
var lstream = require('lstream');

var pipeline = splicer.obj([
    'deps', [ deps() ],
    'pack', [ pack({ raw: true }) ]
]);

pipeline.get('deps').unshift(lstream());

pipeline.get('deps').push(through.obj(function (row, enc, next) {
    row.source = row.source.toUpperCase();
    this.push(row);
    next();
}));

process.stdin.pipe(pipeline).pipe(process.stdout);

Here the deps sub-pipeline is augmented with a post-transformation that uppercases its source input.

methods

var splicer = require('labeled-stream-splicer')

The API is the same as stream-splicer, except that pipeline.get(), pipeline.splice(), and pipeline.indexOf() can accept string labels in addition to numeric indexes.

var pipeline = splicer(streams, opts)

Create a pipeline duplex stream given an array of streams. Each stream will be piped to the next. Writes to pipeline get written to the first stream and data for reads from pipeline come from the last stream.

To signify a label, a stream may have a .label property or a string may be placed in the streams array.

For example, for streams [ a, 'foo', b, c, 'bar', d ], this pipeline is constructed internally:

a.pipe(b).pipe(c).pipe(d)

with a label 'foo' that points to b and a label 'bar' that points to d. If a or c has a .label property, that label would be used for addressing.

Input will get written into a. Output will be read from d.

If any of the elements in streams are arrays, they will be converted into nested labeled pipelines. This is useful if you want to expose a hookable pipeline with grouped insertion points.

var pipeline = splicer.obj(streams, opts)

Create a pipeline with opts.objectMode set to true for convenience.

var removed = pipeline.splice(index, howMany, stream, ...)

Splice the pipeline starting at index, removing howMany streams and replacing them with each additional stream argument provided.

The streams that were removed from the splice and returned.

index can be an integer index or a label.

pipeline.push(stream, ...)

Push one or more streams to the end of the pipeline.

The stream arguments may have a label property that will be used for string lookups.

var stream = pipeline.pop()

Pop a stream from the end of the pipeline.

pipeline.unshift(stream, ...)

Unshift one or more streams to the begining of the pipeline.

The stream arguments may have a label property that will be used for string lookups.

var stream = pipeline.shift()

Shift a stream from the begining of the pipeline.

var stream = pipeline.get(index)

Return the stream at index index.

index can be an integer or a string label.

install

With npm do:

npm install labeled-stream-splicer

license

MIT