Xtreamer
![David](https://img.shields.io/david/dev/manojc/xtreamer.svg)
V1.1.0
- Support for JSON transformer option (demo)
V1.0.0
- Handling to avoid repeating nodes in comments & cdata.
- Minor bug fixes and exception handling.
- Unit tests to cover corner case scenarios.
- No breaking changes.
Background
While working on a NodeJs project, I came across a problem of processing huge XML files. Node process can store a limited amout of data after which it throws out of memory
exception. Thus files more than a specified limit was a pain for the application.
That is where I found a couple of npm packages
viz. xml-flow & xml-stream. These packages process the XML files in chunks using streams (which is exactly what I was looking for).
However, I came across few drawbacks regarding these packages explained in this stackoverflow question. And thus I started working on this package which does exactly what these 2 packages do but it emits xml instead of processed JSON and gives a flexibility of choosing most compatible xml to json npm packages
.
Why Xtreamer ?
-
One of the key points is xtreamer
has stream
as its only dependency.
-
As this package uses streams, the memory consumption will always be in control.
-
xtreamer
itself is an extension of a Transform Stream
so it can be piped to any input Readable Stream
and xtreamer
output can be piped to a Writable Stream
.
-
Apart from above points, xtreamer
provides XML nodes in response which enables it to get hooked up with any XML-JSON
parsing npm packages as per requirement.
Install Package
npm i xtreamer --save
APIs
xtreamer
extends Transform Steram Class & provides an additional custom event xmldata
which emits xml node.
xtreamer( node : string
, options?: object
) : stream.Transform
xtreamer
function initialises the transform stream. It accepts following 2 arguments -
node
: string
options
: object
(coming soon)
This function return a transform stream which can be triggered by piping it with any readable stream.
xmldata event
Apart from default steam events, streamer
emits xmldata
event to emit individual xml nodes.
From version 0.1.3
onwards, xtreamer
also supports the conventional data
event to emit individual xml nodes.
const xtreamer = require("xtreamer");
const xtreamerTransform = xtreamer("XmlNode", options);
xtreamerTransform.on("xmldata", (data) => { });
xtreamerTransform.on("data", (data) => { });
Options
max_xml_size ( default - 10000000 )
max_xml_size
is maximum the number of characters allowed to hold in memory.
xtreamer
raises an error
event in case in memory xml string exceed specified limit. This ensures that the node process doesn't get terminated because of excess in memory data collection.
Default value of this option restricts the amount of data held in memory to approximately 10Mb. Following snippet shows how to override default value -
const xtreamer = require("xtreamer");
const options = { max_xml_size: 30000000 };
const xtreamerTransform = xtreamer("XmlNode", options);
Typically this value is not needed to override as in most of the cases, size of xml in a single xml node will not exceed 10Mb.
transformer ( function(xmlString): jsonObj )
This feature is introduced in version 1.1.0
The transformer
option allows to transform the xml node output to desired JSON strucure by hooking it up with the streaming pipeline. It becomes a very useful feature where JSON parser can be dynamically injected in xtreamer
.
This function is supposed to accept an xml string as a parameter and should return converted valid JSON object in response. The transformer function can also return a promise which will be resolved by xtreamer
before emitting data
event.
Note that the converted JSON is internally stringified before sending it back to in data event handler. So it is advised that the transformer function should always return valid JSON object in response.
In case transformer function encounters an error, xtreamer
emits error
event and stops the xml conversion process.
Usage
Following code snippet uses request
NPM package as input readable stream -
const request = require("request");
const xtreamer = require("xtreamer");
const sampleNode = "SampleNode";
const sampleUrl = "http://sample-xml.com/sample.xml";
let count = 0;
const readStream = request.get(sampleUrl);
const xtreamerTransform = xtreamer(sampleNode)
.on("data", () => ++count % 100 || console.log(count))
.on("end", () => console.log(count))
.on("error", (error) => console.error(error));
readStream.pipe(xtreamerTransform);
As streamer
is a transform stream, one can also pipe the stream with other streams -
const { Writable } = require("stream");
const request = require("request");
const xtreamer = require('xtreamer');
class XtreamerClient extends Writable {
_write(chunk, encoding, next) {
next();
}
}
const sampleNode = "SampleNode";
const sampleUrl = "http://sample-xml.com/sample.xml";
const readStream = request.get(sampleUrl);
const xtreamerTransform = xtreamer(sampleNode)
.on("error", (error) => console.error(error));
readStream.pipe(xtreamerTransform).pipe(new XtreamerClient());
Check the demo for more examples which includes -
-
demo.js - emits xml nodes
-
transformer-demo.js - emits stringified JSON & uses xml-js
package within transformer function
-
transformer-promise-demo.js - emits stringified JSON & uses xml-js
package within transformer function
that return a promise.
Demo
npm i && npm start
Test
npm i && npm run test
Author