ldjson-stream

What is ldjson-stream?

The ldjson-stream npm package is designed to handle line-delimited JSON (LDJSON) streams. It provides utilities for parsing and stringifying LDJSON data, which is useful for processing large datasets in a memory-efficient manner.

What are ldjson-stream's main functionalities?

Parsing LDJSON

This feature allows you to parse a stream of line-delimited JSON data. The code sample reads from a file named 'data.ldjson' and parses each line as a JSON object, logging each object to the console.

const ldjson = require('ldjson-stream');
const fs = require('fs');

fs.createReadStream('data.ldjson')
  .pipe(ldjson.parse())
  .on('data', obj => {
    console.log(obj);
  });

Stringifying LDJSON

This feature allows you to stringify a stream of JSON objects into line-delimited JSON format. The code sample takes an array of objects and writes them to a file named 'output.ldjson' in LDJSON format.

const ldjson = require('ldjson-stream');
const fs = require('fs');

const data = [
  { name: 'Alice', age: 30 },
  { name: 'Bob', age: 25 }
];

const writeStream = fs.createWriteStream('output.ldjson');
const stringifyStream = ldjson.serialize();

stringifyStream.pipe(writeStream);
data.forEach(item => stringifyStream.write(item));
stringifyStream.end();

Other packages similar to ldjson-stream

JSONStream

JSONStream is a popular package for working with streaming JSON data. It provides similar functionality to ldjson-stream, including parsing and stringifying JSON data. However, JSONStream is more general-purpose and can handle various JSON formats, not just line-delimited JSON.

ndjson

ndjson is another package specifically designed for handling newline-delimited JSON streams. It offers similar features to ldjson-stream, such as parsing and stringifying NDJSON data. The main difference is in the API design and implementation details.

event-stream

event-stream is a toolkit for working with streams in Node.js. It includes utilities for parsing and stringifying JSON data, among other stream-related functionalities. While it is not focused solely on LDJSON, it can be used to achieve similar results.

README

ldjson-stream

streaming line delimited json parser + serializer

usage

var ldj = require('ldjson-stream')

ldj.parse()

returns a transform stream that accepts newline delimited json and emits objects

example newline delimited json:

data.txt:

{"foo": "bar"}
{"hello": "world"}

If you want to discard non-valid JSON messages, you can call ldj.parse({strict: false})

usage:

fs.createReadStream('data.txt')
  .pipe(ldj.parse())
  .on('data', function(obj) {
    // obj is a javascript object
  })

ldj.serialize()

returns a transform stream that accepts json objects and emits newline delimited json

example usage:

var serialize = ldj.serialize()
serialize.on('data', function(line) {
  // line is a line of stringified JSON with a newline delimiter at the end
})
serialize.write({"foo": "bar"})
serialize.end()

license

BSD