@e2fyi/streams

@e2fyi/streams

This NodeJS library provides custom NodeJS streams for specific use cases.

Currently, the following streams are available:

• DocumentTagger: A Transform stream to tag an auto-increment field to each object in the stream. Can also mutate the stream objects through a function or default Object.

• MongooseStream: A Transform stream which will bulk write the objects in the stream to mongodb via a mongoose model.

ChangeLog

• v1.0.0: new streams: DocumentTagger, MongooseStream.

Quick start

Using CommonJS module

// importing DocumentTagger and MongooseStream
const {DocumentTagger, MongooseStream} = require('@e2fyi/streams');

API reference

The API documentation is also available at https://e2fyi.github.io/streams.

@e2fyi/streams

• @e2fyi/streams
  • .DocumentTagger ⇐ stream.Transform
    • new DocumentTagger(opts)
  • .MongooseStream ⇐ stream.Transform
    • new MongooseStream(opts)
  • .DocumentTaggerSettings : Object
  • .MongooseStreamSettings : Object

@e2fyi/streams.DocumentTagger ⇐ stream.Transform

Transform Object stream (objectMode=true) to tag with an autoIncrement id. An object or function can be optionally provided to mutate each object in the stream.

Kind: static class of @e2fyi/streams
Extends: stream.Transform
Emits: filtered

new DocumentTagger(opts)

Create a new DocumentTagger stream.

Param	Type	Description
opts	DocumentTaggerSettings	Settings for the stream.

Example

const docTagger = new DocumentTagger({autoIncrement: 'id', mutate: { project: 'test' }});
someReadableStreamFromArray([{text: 'abc'}, {text: 'efg'}])
  .pipe(docTagger)
  .pipe(process.stdout);
// stdout >
// {"text": "abc", "id": 0, "project": "test"}
// {"text": "efg", "id": 1, "project": "test"}

@e2fyi/streams.MongooseStream ⇐ stream.Transform

A custom NodeJS Transform stream to mongo via mongoose.

Kind: static class of @e2fyi/streams
Extends: stream.Transform
Emits: mongoose-bulk-write

new MongooseStream(opts)

Create a Transform stream which bulkWrite to mongo based on the itemWaterMark. model (mongoose Model) is a required field.

Param	Type	Description
opts	MongooseStreamSettings	Configuration for MongoStream. Default value for itemWaterMark is 50.

Example

var stream2mongo = new MongooseStream({mode: SomeMongooseModel});
someReadableStreamFromArray([{text: 'abc'}, {text: 'efg'}])
  .pipe(stream2mongo) // writes to mongo (while stream are also passthrough)
  .pipe(response); // stream same results back to some request

@e2fyi/streams.DocumentTaggerSettings : Object

Settings for DocumentTagger.

Kind: static typedef of @e2fyi/streams
Properties

Name	Type	Description
autoIncrement	String	The auto-increment field to tag onto the stream object.
ignoreNonObject	Boolean	If true, no errors will be emitted when the chunk in the stream cannot be parsed into Object.
readableObjectMode	Boolean	If true, Object will be emitted from the stream, otherwise a String or Buffer representation or will emitted instead.
objectMode	Boolean	If true, Object will be emitted from the stream, otherwise a String or Buffer representation or will emitted instead. Overwrites writableObjectMode.
filter	function	Function to filter the objects in the stream. Return true to keep the object.
mutate	function | Object	Function or Object to mutate the stream. If an Object is provided, each stream object will be mutated with the Object.assign(streamObj, mutateObj).

@e2fyi/streams.MongooseStreamSettings : Object

Settings for MongooseStream.

Kind: static typedef of @e2fyi/streams
Properties

Name	Type	Default	Description
itemWaterMark	Number	50	The number of item collected before writing to mongodb.
passThrough	Boolean		If false nothing will be emitted from the stream.
model	mongoose.Model		mongoose Model.

Development

Unit testing

npm test

Build documentation

npm run build