minipass

What is minipass?

The minipass npm package is a small, simple stream.PassThrough class. It is designed to be a minimal implementation of a streaming PassThrough, which is a type of Duplex stream that reads from a readable source and writes to a writable destination with minimal overhead. It is useful for cases where you want to collect stream data, transform it, or simply pass it through unmodified.

What are minipass's main functionalities?

Basic Stream Collection

This feature allows you to collect data from a stream. The 'data' event is emitted whenever the stream has data available. The 'write' method is used to send data into the stream, and 'end' is used to signal that no more data will be written.

const MiniPass = require('minipass')
const stream = new MiniPass()
stream.on('data', chunk => {
  console.log('Got some data:', chunk.toString())
})
stream.write('hello')
stream.end('world')

Piping Data

This feature demonstrates how to pipe data from a MiniPass stream to another writable stream. In this example, data is piped to a file stream, which writes the data to 'output.txt'.

const MiniPass = require('minipass')
const fs = require('fs')
const stream = new MiniPass()
const writable = fs.createWriteStream('output.txt')
stream.pipe(writable)
stream.write('hello')
stream.end('world')

Transforming Stream Data

This feature shows how to extend MiniPass to create a custom transform stream. In this example, an Uppercase class is created that converts all incoming data to uppercase before passing it through.

const MiniPass = require('minipass')
class Uppercase extends MiniPass {
  write (chunk, encoding, callback) {
    super.write(chunk.toString().toUpperCase(), encoding, callback)
  }
}
const ucStream = new Uppercase()
ucStream.on('data', chunk => {
  console.log(chunk.toString())
})
ucStream.write('hello')
ucStream.end('world')

Other packages similar to minipass

through2

Through2 is a tiny wrapper around Node streams.Transform, making it easy to create transform streams. It is similar to minipass in that it provides a simple way to handle stream data, but it has a slightly different API and additional convenience methods.

pumpify

Pumpify combines an array of streams into a single duplex stream. It is similar to minipass in that it deals with stream data, but it focuses on combining streams rather than simply passing data through.

README

minipass

A very minimal implementation of a PassThrough stream

It's very fast for objects, strings, and buffers.

Supports pipe()ing (including multi-pipe() and backpressure transmission), buffering data until either a data event handler or pipe() is added (so you don't lose the first chunk), and most other cases where PassThrough is a good idea.

There is a read() method, but it's much more efficient to consume data from this stream via 'data' events or by calling pipe() into some other stream. Calling read() requires the buffer to be flattened in some cases, which requires copying memory.

There is also no unpipe() method. Once you start piping, there is no stopping it!

If you set objectMode: true in the options, then whatever is written will be emitted. Otherwise, it'll do a minimal amount of Buffer copying to ensure proper Streams semantics when read(n) is called.

This is not a through or through2 stream. It doesn't transform the data, it just passes it right through. If you want to transform the data, extend the class, and override the write() method. Once you're done transforming the data however you want, call super.write() with the transform output.

For an example of a stream that extends MiniPass to provide transform capabilities, check out minizlib.

USAGE

const MiniPass = require('minipass')
const mp = new MiniPass(options) // optional: { encoding }
mp.write('foo')
mp.pipe(someOtherStream)
mp.end('bar')

collecting

mp.collect().then(all => {
  // all is an array of all the data emitted
  // encoding is supported in this case, so
  // so the result will be a collection of strings if
  // an encoding is specified, or buffers/objects if not.
  //
  // In an async function, you may do
  // const data = await stream.collect()
})

collecting into a single blob

This is a bit slower because it concatenates the data into one chunk for you, but if you're going to do it yourself anyway, it's convenient this way:

mp.concat().then(onebigchunk => {
  // onebigchunk is a string if the stream
  // had an encoding set, or a buffer otherwise.
})

iteration

You can iterate over streams synchronously or asynchronously in platforms that support it.

Synchronous iteration will end when the currently available data is consumed, even if the end event has not been reached. In string and buffer mode, the data is concatenated, so unless multiple writes are occurring in the same tick as the read(), sync iteration loops will generally only have a single iteration.

To consume chunks in this way exactly as they have been written, with no flattening, create the stream with the { objectMode: true } option.

const mp = new Minipass({ objectMode: true })
mp.write('a')
mp.write('b')
for (let letter of mp) {
  console.log(letter) // a, b
}
mp.write('c')
mp.write('d')
for (let letter of mp) {
  console.log(letter) // c, d
}
mp.write('e')
mp.end()
for (let letter of mp) {
  console.log(letter) // e
}
for (let letter of mp) {
  console.log(letter) // nothing
}

Asynchronous iteration will continue until the end event is reached, consuming all of the data.

const mp = new Minipass({ encoding: 'utf8' })

// some source of some data
let i = 5
const inter = setInterval(() => {
  if (i --> 0)
    mp.write(Buffer.from('foo\n', 'utf8'))
  else {
    mp.end()
    clearInterval(inter)
  }
}, 100)

// consume the data with asynchronous iteration
async function consume () {
  for await (let chunk of mp) {
    console.log(chunk)
  }
  return 'ok'
}

consume().then(res => console.log(res))
// logs `foo\n` 5 times, and then `ok`