Security News
Research
Data Theft Repackaged: A Case Study in Malicious Wrapper Packages on npm
The Socket Research Team breaks down a malicious wrapper package that uses obfuscation to harvest credentials and exfiltrate sensitive data.
A tiny wrapper around Node.js streams.Transform (Streams2/3) to avoid explicit subclassing noise
The through2 npm package is a thin wrapper around Node.js streams.Transform (a standard Node.js API) that makes it easier to create transform streams. It is designed to work with streams in object mode or non-object mode, allowing for simple function-based stream transformation. This package is particularly useful when you need to create a custom stream that modifies or transforms data as it passes through the stream pipeline.
Simple Transform
This feature allows you to create a simple transform stream that converts input data to uppercase. The code sample demonstrates how to use through2 to create a stream that reads from stdin, transforms the data, and writes to stdout.
const through2 = require('through2');
const stream = through2(function(chunk, enc, callback) {
this.push(chunk.toString().toUpperCase());
callback();
});
process.stdin.pipe(stream).pipe(process.stdout);
Object Mode Transform
This feature allows you to work with streams in object mode, where the stream chunks are JavaScript objects. The code sample shows how to create a transform stream that modifies properties of objects and how to use it with a readable stream of objects.
const through2 = require('through2').obj;
const stream = through2(function(obj, enc, callback) {
obj.key = obj.key.toUpperCase();
this.push(obj);
callback();
});
// Usage with an array of objects
const objects = [{ key: 'value' }, { key: 'data' }];
const objectStream = require('stream').Readable.from(objects);
objectStream.pipe(stream).on('data', (transformedObj) => console.log(transformedObj));
Flush Function
This feature allows you to perform an action when the stream is ending, just before it finishes. The code sample demonstrates how to append a message to the stream output right before the stream ends.
const through2 = require('through2');
const stream = through2(function(chunk, enc, callback) {
this.push(chunk);
callback();
}, function(callback) {
// This function is called before the stream is finished
this.push('Stream is ending!');
callback();
});
process.stdin.pipe(stream).pipe(process.stdout);
The 'through' package provides a simple way to create a readable-writable stream. It is similar to through2 but does not support object mode by default and has a less modern API.
The 'stream-transform' package is part of the Node.js CSV project and provides a standard transform stream with a simpler API for synchronous operations. It is specifically designed for CSV data transformation and is less generic compared to through2.
The 'pumpify' package combines an array of streams into a single duplex stream using pump and duplexify. It is similar to through2 in that it deals with stream transformation but focuses on combining streams rather than creating individual transform streams.
A tiny wrapper around Node.js streams.Transform (Streams2/3) to avoid explicit subclassing noise
Inspired by Dominic Tarr's through in that it's so much easier to make a stream out of a function than it is to set up the prototype chain properly: through(function (chunk) { ... })
.
fs.createReadStream('ex.txt')
.pipe(through2(function (chunk, enc, callback) {
for (let i = 0; i < chunk.length; i++)
if (chunk[i] == 97)
chunk[i] = 122 // swap 'a' for 'z'
this.push(chunk)
callback()
}))
.pipe(fs.createWriteStream('out.txt'))
.on('finish', () => doSomethingSpecial())
Or object streams:
const all = []
fs.createReadStream('data.csv')
.pipe(csv2())
.pipe(through2.obj(function (chunk, enc, callback) {
const data = {
name : chunk[0]
, address : chunk[3]
, phone : chunk[10]
}
this.push(data)
callback()
}))
.on('data', (data) => {
all.push(data)
})
.on('end', () => {
doSomethingSpecial(all)
})
Note that through2.obj(fn)
is a convenience wrapper around through2({ objectMode: true }, fn)
.
Since Node.js introduced Simplified Stream Construction, many uses of through2 have become redundant. Consider whether you really need to use through2 or just want to use the 'readable-stream'
package, or the core 'stream'
package (which is derived from 'readable-stream'
):
const { Transform } = require('readable-stream')
const transformer = new Transform({
transform(chunk, enc, callback) {
// ...
}
})
through2([ options, ] [ transformFunction ] [, flushFunction ])
Consult the stream.Transform documentation for the exact rules of the transformFunction
(i.e. this._transform
) and the optional flushFunction
(i.e. this._flush
).
The options argument is optional and is passed straight through to stream.Transform
. So you can use objectMode:true
if you are processing non-binary streams (or just use through2.obj()
).
The options
argument is first, unlike standard convention, because if I'm passing in an anonymous function then I'd prefer for the options argument to not get lost at the end of the call:
fs.createReadStream('/tmp/important.dat')
.pipe(through2({ objectMode: true, allowHalfOpen: false },
(chunk, enc, cb) => {
cb(null, 'wut?') // note we can use the second argument on the callback
// to provide data as an alternative to this.push('wut?')
}
))
.pipe(fs.createWriteStream('/tmp/wut.txt'))
The transformFunction
must have the following signature: function (chunk, encoding, callback) {}
. A minimal implementation should call the callback
function to indicate that the transformation is done, even if that transformation means discarding the chunk.
To queue a new chunk, call this.push(chunk)
—this can be called as many times as required before the callback()
if you have multiple pieces to send on.
Alternatively, you may use callback(err, chunk)
as shorthand for emitting a single chunk or an error.
If you do not provide a transformFunction
then you will get a simple pass-through stream.
The optional flushFunction
is provided as the last argument (2nd or 3rd, depending on whether you've supplied options) is called just prior to the stream ending. Can be used to finish up any processing that may be in progress.
fs.createReadStream('/tmp/important.dat')
.pipe(through2(
(chunk, enc, cb) => cb(null, chunk), // transform is a noop
function (cb) { // flush function
this.push('tacking on an extra buffer to the end');
cb();
}
))
.pipe(fs.createWriteStream('/tmp/wut.txt'));
through2.ctor([ options, ] transformFunction[, flushFunction ])
Instead of returning a stream.Transform
instance, through2.ctor()
returns a constructor for a custom Transform. This is useful when you want to use the same transform logic in multiple instances.
const FToC = through2.ctor({objectMode: true}, function (record, encoding, callback) {
if (record.temp != null && record.unit == "F") {
record.temp = ( ( record.temp - 32 ) * 5 ) / 9
record.unit = "C"
}
this.push(record)
callback()
})
// Create instances of FToC like so:
const converter = new FToC()
// Or:
const converter = FToC()
// Or specify/override options when you instantiate, if you prefer:
const converter = FToC({objectMode: true})
through2 is Copyright © Rod Vagg and additional contributors and licensed under the MIT license. All rights not explicitly granted in the MIT license are reserved. See the included LICENSE file for more details.
FAQs
A tiny wrapper around Node.js streams.Transform (Streams2/3) to avoid explicit subclassing noise
We found that through2 demonstrated a not healthy version release cadence and project activity because the last version was released a year ago. It has 2 open source maintainers collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
Research
The Socket Research Team breaks down a malicious wrapper package that uses obfuscation to harvest credentials and exfiltrate sensitive data.
Research
Security News
Attackers used a malicious npm package typosquatting a popular ESLint plugin to steal sensitive data, execute commands, and exploit developer systems.
Security News
The Ultralytics' PyPI Package was compromised four times in one weekend through GitHub Actions cache poisoning and failure to rotate previously compromised API tokens.