Security News
OpenJS: “XZ Utils Cyberattack Likely Not an Isolated Incident”
OpenJS is warning of social engineering takeovers targeting open source projects after receiving a credible attempt on the foundation.
stream-json
Advanced tools
Weekly downloads
Readme
stream-json
is a micro-library of node.js stream components with minimal dependencies for creating custom data processors oriented on processing huge JSON files while requiring a minimal memory footprint. It can parse JSON files far exceeding available memory. Even individual primitive data items (keys, strings, and numbers) can be streamed piece-wise. Streaming SAX-inspired event-based API is included as well.
Available components:
Pick
, or generated by other means."{}[]"
), or with white spaces (like in "true 1 null"
).utf8
text input.StreamValues
.
Parser({jsonStreaming: true})
+ StreamValues
.Disassembler
+ Stringer
.All components are meant to be building blocks to create flexible custom data processing pipelines. They can be extended and/or combined with custom code. They can be used together with stream-chain to simplify data processing.
This toolkit is distributed under New BSD license.
const {chain} = require('stream-chain');
const {parser} = require('stream-json');
const {pick} = require('stream-json/filters/Pick');
const {ignore} = require('stream-json/filters/Ignore');
const {streamValues} = require('stream-json/streamers/StreamValues');
const fs = require('fs');
const zlib = require('zlib');
const pipeline = chain([
fs.createReadStream('sample.json.gz'),
zlib.createGunzip(),
parser(),
pick({filter: 'data'}),
ignore({filter: /\b_meta\b/i}),
streamValues(),
data => {
const value = data.value;
// keep data only for the accounting department
return value && value.department === 'accounting' ? data : null;
}
]);
let counter = 0;
pipeline.on('data', () => ++counter);
pipeline.on('end', () =>
console.log(`The accounting department has ${counter} employees.`));
See the full documentation in Wiki.
Companion projects:
stream-json
:
rows as arrays of string values. If a header row is used, it can stream rows as objects with named fields.npm install --save stream-json
# or: yarn add stream-json
The whole library is organized as a set of small components, which can be combined to produce the most effective pipeline. All components are based on node.js streams, and events. They implement all required standard APIs. It is easy to add your own components to solve your unique tasks.
The code of all components is compact and simple. Please take a look at their source code to see how things are implemented, so you can produce your own components in no time.
Obviously, if a bug is found, or a way to simplify existing components, or new generic components are created, which can be reused in a variety of projects, don't hesitate to open a ticket, and/or create a pull request.
stream-chain
), bugfix: inconsistent object/array braces. Thx Xiao Li.utils/Utf8Stream
to sanitize utf8
input, all parsers support it automatically. Thx john30 for the suggestion.jsonl/Parser
and jsonl/Stringer
.The rest can be consulted in the project's wiki Release history.
FAQs
stream-json is the micro-library of Node.js stream components for creating custom JSON processing pipelines with a minimal memory footprint. It can parse JSON files far exceeding available memory streaming individual primitives using a SAX-inspired API. I
The npm package stream-json receives a total of 867,705 weekly downloads. As such, stream-json popularity was classified as popular.
We found that stream-json demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
OpenJS is warning of social engineering takeovers targeting open source projects after receiving a credible attempt on the foundation.
Company News
Come meet the Socket team at BSidesSF and RSA! We're sponsoring several fun networking events and we would love to see you there.
Security News
OSI is starting a conversation aimed at removing the excuse of the SaaS loophole for companies navigating licensing and the complexities of doing business with open source.