What is big-json?
The big-json npm package is designed to handle large JSON files efficiently. It provides functionality to parse and stringify large JSON objects without running into memory issues, making it suitable for applications that need to process big data in JSON format.
What are big-json's main functionalities?
Parse large JSON files
This feature allows you to parse large JSON files by streaming the data, which helps in avoiding memory overflow issues. The code sample demonstrates how to create a read stream from a large JSON file and pipe it into a parse stream provided by big-json.
const fs = require('fs');
const bigJson = require('big-json');
const readStream = fs.createReadStream('path/to/large-file.json');
const parseStream = bigJson.createParseStream();
parseStream.on('data', (data) => {
console.log('Parsed data:', data);
});
readStream.pipe(parseStream);
Stringify large JSON objects
This feature allows you to stringify large JSON objects and write them to a file in a memory-efficient manner. The code sample shows how to create a stringify stream from a large JSON object and pipe it into a write stream to save it to a file.
const fs = require('fs');
const bigJson = require('big-json');
const writeStream = fs.createWriteStream('path/to/output-file.json');
const stringifyStream = bigJson.createStringifyStream({
json: { large: 'object' }
});
stringifyStream.pipe(writeStream);
Other packages similar to big-json
json-stream
The json-stream package provides similar functionality for parsing JSON data in a streaming fashion. It is useful for handling large JSON files without loading the entire file into memory. Compared to big-json, json-stream focuses more on parsing rather than stringifying large JSON objects.
stream-json
The stream-json package offers a comprehensive set of tools for working with JSON in a streaming manner. It supports both parsing and stringifying large JSON files. Stream-json is more modular and provides finer control over the streaming process compared to big-json.
JSONStream
JSONStream is another package that allows for streaming JSON parsing and stringifying. It is widely used and well-documented, making it a popular choice for handling large JSON files. JSONStream provides a flexible API for working with JSON data in a streaming context, similar to big-json.
big-json
A stream based implementation of JSON.parse and JSON.stringify for big POJOs
There exist many stream based implementations of JSON parsing or stringifying
for large data sets. These implementations typical target time series data, new
line delimited data or other array-like data, e.g., logging records or other
continuous flowing data.
This module hopes to fill a gap in the ecosystem: parsing large JSON objects
that are just really big objects. With large in-memory objects, it is
possible to run up against the V8 string length limitation, which is currently
(as of 9/2017) limited to 512MB. Thus, if your large object has enough keys
or values, it is possible to exceed the string length limit when calling
JSON.stringify.
Similarly, when retrieving stored JSON from disk or over the network, if the
JSON stringified representation of the object exceeds the string length limit,
the process will throw when attempting to convert the Buffer into a string.
The only way to work with such large objects is to use a streaming
implementation of both JSON.parse
and JSON.stringify
. This module does just
that by normalizing the APIs for different modules that have previously
published, combining both parse and stringify functions into a single module.
These underlying modules are subject to change at anytime.
The major caveat is that the reconstructed POJO must be able to fit in memory.
If the reconstructed POJO cannot be stored in memory, then it may be time to
reconsider the way these large objects are being transported and processed.
This module currently uses
JSONStream for parsing, and
json-stream-stringify for
stringification.
Getting Started
Install the module with: npm install big-json
Usage
To parse a big JSON coming from an external source:
const fs = require('fs');
const path = require('path');
const json = require('big-json');
const readStream = fs.createReadStream('big.json');
const parseStream = json.createParseStream();
parseStream.on('data', function(pojo) {
});
readStream.pipe(parseStream);
To stringify JSON:
const json = require('big-json');
const stringifyStream = json.createStringifyStream({
body: BIG_POJO
});
stringifyStream.on('data', function(strChunk) {
});
API
createParseStream()
Parses an incoming stream and accumulates it into a POJO. Supports both objects
and arrays as root objects for stream data.
Returns: {Stream} a JSON.parse stream
createStringifyStream(opts)
opts
{Object} an options objectopts.body
{Object | Array} an object or array to JSON.stringify
Returns: {Stream} a JSON.stringify stream
parse(opts, [callback])
An async JSON.parse using the same underlying stream implementation. If a
callback is not passed, a promise is returned.
opts
{Object} an options objectopts.body
{String | Buffer} the string or buffer to be parsedcallback
{Function} a callback object
Returns: {Object | Array} the parsed JSON
stringify(opts, [callback])
An async JSON.stringify using the same underlying stream implementation. If a
callback is not passed, a promise is returned.
opts
{Object} an options objectopts.body
{Object} the object to be stringifiedcallback
{Function} a callback object
Returns: {Object} the stringified object
Contributing
Ensure that all linting and codestyle tasks are passing. Add unit tests for any
new or changed functionality.
To start contributing, install the git prepush hooks:
make githooks
Before committing, lint and test your code using the included Makefile:
make prepush
License
Copyright (c) 2019 Alex Liu
Licensed under the MIT license.