Huge News!Announcing our $40M Series B led by Abstract Ventures.Learn More →
stream-json
Advanced tools
stream-json - npm Package Compare versions
Comparing version 0.4.2 to 0.5.0
11
Combo.js
@@ -14,5 +14,6 @@ "use strict"; | ||
| if(options){ | ||
| this._packKeys = options.packKeys; | ||
| this._packStrings = options.packStrings; | ||
| this._packNumbers = options.packNumbers; | ||
| this._packKeys = options.packKeys; | ||
| this._packStrings = options.packStrings; | ||
| this._packNumbers = options.packNumbers; | ||
| this._jsonStreaming = options.jsonStreaming; | ||
| } | ||
@@ -473,2 +474,6 @@ | ||
| if(this._buffer){ | ||
| if(this._jsonStreaming){ | ||
| this._expect = "value"; | ||
| break; | ||
| } | ||
| return callback(new Error("Parser cannot parse input: unexpected characters")); | ||
@@ -475,0 +480,0 @@ } |
| { | ||
| "name": "stream-json", | ||
| "version": "0.4.2", | ||
| "version": "0.5.0", | ||
| "description": "stream-json is a SAX-inspired stream components with a minimal memory footprint to parse huge JSON files. Includes utilities to stream Django-like JSON database dumps.", | ||
@@ -5,0 +5,0 @@ "homepage": "http://github.com/uhop/stream-json", |
@@ -18,2 +18,3 @@ "use strict"; | ||
| this._parent = ""; | ||
| this._jsonStreaming = options && options.jsonStreaming; | ||
| } | ||
@@ -420,2 +421,6 @@ util.inherits(Parser, Transform); | ||
| if(this._buffer){ | ||
| if(this._jsonStreaming){ | ||
| this._expect = "value"; | ||
| break; | ||
| } | ||
| return callback(new Error("Parser cannot parse input: unexpected characters")); | ||
@@ -422,0 +427,0 @@ } |
@@ -75,4 +75,6 @@ # stream-json | ||
| `options` can contain some technical parameters, and it is rarely needs to be specified. You can find it thoroughly documented in [node.js' Stream documentation](http://nodejs.org/api/stream.html). | ||
| `options` can contain some technical parameters, and it rarely needs to be specified. You can find it thoroughly documented in [node.js' Stream documentation](http://nodejs.org/api/stream.html). Additionally it recognizes following properties: | ||
| * `jsonStreaming` can be `true` or `false` (the default). If `true`, it can parse a stream of JSON objects as described in [JSON Streaming](https://en.wikipedia.org/wiki/JSON_Streaming) as "Concatenated JSON". Technically it will recognize "Line delimited JSON" as well. | ||
| The test files for `Parser`: `tests/test_parser.js`, `tests\manual\test_parser.js`. Actually all test files in `tests/` use `Parser`. | ||
@@ -182,3 +184,3 @@ | ||
| `Combo` is a [Transform](https://nodejs.org/api/stream.html#stream_class_stream_transform) stream, which combines `Parser`, `Streamer`, and `Packer`. It accepts the same extra options as `Packer`, and produces a stream of objects expected from `Streamer`, and augmented by `Packer`. Please refer to the documentation of those three components for more details. | ||
| `Combo` is a [Transform](https://nodejs.org/api/stream.html#stream_class_stream_transform) stream, which combines `Parser`, `Streamer`, and `Packer`. It accepts the same extra options as `Parser` and `Packer`, and produces a stream of objects expected from `Streamer`, and augmented by `Packer`. Please refer to the documentation of those three components for more details. | ||
@@ -292,3 +294,3 @@ While logically `Combo` is a combination of three existing components, it has an important advantage: speed. The codes for `Parser`, `Streamer`, and `Packer` are merged together in one component avoiding overhead of node.js streams completely, which is significant. It is recommended to use `Combo` over a chain of `Parser` + `Streamer`, or `Parser` + `Streamer` + `Packer`. | ||
| * `input` — the beginning of a pipeline, which should be used as an input for a JSON stream. | ||
| * `output` — the end of a pipeline, which can be used to pipe the resulting stream of objects for futher processing. | ||
| * `output` — the end of a pipeline, which can be used to pipe the resulting stream of objects for further processing. | ||
@@ -334,2 +336,4 @@ The test files for `Source`: `tests/test_source.js` and `tests/manual/test_source.js`. | ||
| It doesn't support `jsonStreaming` option. | ||
| The test file for `ClassicParser`: `tests/test_classic.js`. | ||
@@ -343,2 +347,4 @@ | ||
| It doesn't support `jsonStreaming` option. | ||
| The test file for `AltParser`: `tests/test_alternative.js`. | ||
@@ -395,3 +401,3 @@ | ||
| It is a [Transform](https://nodejs.org/api/stream.html#stream_class_stream_transform) stream, which opertes in an [objectMode](http://nodejs.org/api/stream.html#stream_object_mode). | ||
| It is a [Transform](https://nodejs.org/api/stream.html#stream_class_stream_transform) stream, which operates in an [objectMode](http://nodejs.org/api/stream.html#stream_object_mode). | ||
@@ -428,3 +434,3 @@ `StreamArray` produces a stream of objects in following format: | ||
| * `input` — the beginning of a pipeline, which should be used as an input for a JSON stream. | ||
| * `output` — the end of a pipeline, which can be used for events, or to pipe the resulting stream of objects for futher processing. | ||
| * `output` — the end of a pipeline, which can be used for events, or to pipe the resulting stream of objects for further processing. | ||
@@ -461,3 +467,3 @@ The test file for `StreamArray`: `tests/test_array.js`. | ||
| It is a [Transform](https://nodejs.org/api/stream.html#stream_class_stream_transform) stream, which opertes in an [objectMode](http://nodejs.org/api/stream.html#stream_object_mode). | ||
| It is a [Transform](https://nodejs.org/api/stream.html#stream_class_stream_transform) stream, which operates in an [objectMode](http://nodejs.org/api/stream.html#stream_object_mode). | ||
@@ -524,2 +530,44 @@ Just like `StreamArray`, `StreamFilteredArray` produces a stream of objects in following format: | ||
| ### utils/StreamJsonObjects | ||
| This utility deals with [JSON Streaming](https://en.wikipedia.org/wiki/JSON_Streaming) -- "Concatenated JSON", when values are sent separated syntactically (e.g., `"{}[]"`) or with whitespaces (e.g., `"true 1 null"`), and "Line delimited JSON". The assumption is that while individual values fit in memory, the stream itself does not. | ||
| This helper is modeled after `utils/StreamArray`. | ||
| It is a [Transform](https://nodejs.org/api/stream.html#stream_class_stream_transform) stream, which operates in [objectMode](http://nodejs.org/api/stream.html#stream_object_mode). | ||
| `StreamJsonObjects` produces a stream of objects in following format: | ||
| ```js | ||
| {index, value} | ||
| ``` | ||
| Where `index` is a numeric (artificial) index in the stream starting from 0, and `value` is a corresponding value. All objects are produced strictly sequentially. | ||
| ```js | ||
| var StreamJsonObjects = require("stream-json/utils/StreamJsonObjects"); | ||
| var stream = StreamJsonObjects.make(); | ||
| // Example of use: | ||
| stream.output.on("data", function(object){ | ||
| console.log(object.index, object.value); | ||
| }); | ||
| stream.output.on("end", function(){ | ||
| console.log("done"); | ||
| }); | ||
| fs.createReadStream(fname).pipe(stream.input); | ||
| ``` | ||
| `StreamJsonObjects` is a constructor, which optionally takes one object: `options`. `options` can contain some technical parameters, and it is rarely needs to be specified. You can find it thoroughly documented in [node.js' Stream documentation](http://nodejs.org/api/stream.html). | ||
| Directly on `StreamJsonObjects` there is a class-level helper function `make()`, which helps to construct a proper pipeline. It is similar to `makeSource()` and takes the same argument `options`. Internally it creates and connects `Combo` and `StreamArray`, and returns an object with three properties: | ||
| * `streams` — an array of streams so you can inspect them individually, if needed. They are connected sequentially in the array order. | ||
| * `input` — the beginning of a pipeline, which should be used as an input for a JSON stream. | ||
| * `output` — the end of a pipeline, which can be used for events, or to pipe the resulting stream of objects for further processing. | ||
| The test file for `StreamJsonObjects`: `tests/test_json_stream.js`. | ||
| ### utils/FilterObjects | ||
@@ -535,3 +583,3 @@ | ||
| It is a [Transform](https://nodejs.org/api/stream.html#stream_class_stream_transform) stream, which opertes in an [objectMode](http://nodejs.org/api/stream.html#stream_object_mode). | ||
| It is a [Transform](https://nodejs.org/api/stream.html#stream_class_stream_transform) stream, which operates in an [objectMode](http://nodejs.org/api/stream.html#stream_object_mode). | ||
@@ -590,2 +638,10 @@ ```js | ||
| ## FAQ | ||
| ### What if my utf-8 data is decoded incorrectly? | ||
| `stream-json` does not decode utf-8 relying on Node to do it correctly. Apparently in some cases Node can fail to decode multi-byte characters correctly, when they are split between different buffers. If you encounter that problem (I did not see it in the wild yet), you can solve it by piping an input stream through a sanitizer before sending it to `stream-json` parser. These two packages look promising, and appear to be doing the right thing: | ||
| * https://www.npmjs.com/package/utf8-stream | ||
| * https://www.npmjs.com/package/utf8-align-stream | ||
| ## Credits | ||
@@ -639,2 +695,3 @@ | ||
| - 0.5.0 *added support for [JSON Streaming](https://en.wikipedia.org/wiki/JSON_Streaming)* | ||
| - 0.4.2 *refreshed dependencies.* | ||
@@ -659,4 +716,4 @@ - 0.4.1 *added `StreamObject` by [Sam Noedel](https://github.com/delta62)* | ||
| [dev-deps-image]: https://img.shields.io/david/dev/uhop/stream-json.svg | ||
| [dev-deps-url]: https://david-dm.org/uhop/stream-json#info=devDependencies | ||
| [dev-deps-url]: https://david-dm.org/uhop/stream-json?type=dev | ||
| [travis-image]: https://img.shields.io/travis/uhop/stream-json.svg | ||
| [travis-url]: https://travis-ci.org/uhop/stream-json |
@@ -23,4 +23,5 @@ "use strict"; | ||
| require("./test_object"); | ||
| require("./test_json_stream"); | ||
| unit.run(); |
Sorry, the diff of this file is not supported yet
New alerts
License Policy Violation
LicenseThis package is not allowed per your license policy. Review the package's license to ensure compliance.
Found 1 instance in 1 package
Fixed alerts
License Policy Violation
LicenseThis package is not allowed per your license policy. Review the package's license to ensure compliance.
Found 1 instance in 1 package
Improved metrics
- Total package byte prevSize
- increased by3.23%
213106
- Number of package files
- increased by5.45%
58
- Lines of code
- increased by2.44%
3742
- Number of lines in readme file
- increased by8.77%
707
Worsened metrics
- Number of medium supply chain risk alerts
- increased by0.42%
240
No dependency changes