stream-json - npm Package Compare versions

Comparing version 0.6.1 to 1.0.0

package.json

 {
   "name": "stream-json",
-  "version": "0.6.1",
-  "description": "stream-json is a SAX-inspired stream components with a minimal memory footprint to parse huge JSON files. Includes utilities to stream Django-like JSON database dumps.",
+  "version": "1.0.0",
+  "description": "stream-json is SAX-inspired stream components with a minimal memory footprint to parse huge JSON files. Includes utilities to stream Django-like JSON database dumps.",
   "homepage": "http://github.com/uhop/stream-json",
   "bugs": "http://github.com/uhop/stream-json/issues",
-  "main": "main.js",
+  "main": "src/main.js",
   "directories": {
@@ -12,3 +12,3 @@ "test": "tests"
   "dependencies": {
-    "parser-toolkit": ">=0.0.3"
+    "stream-chain": "^2.0.2"
   },
@@ -15,0 +15,0 @@ "devDependencies": {

README.md

@@ -9,710 +9,89 @@ # stream-json
 
-`stream-json` is a collection of node.js stream components for creating custom standard-compliant JSON processors, which requires a minimal memory footprint. It can parse JSON files far exceeding available memory. Even individual primitive data items (keys, strings, and numbers) can be streamed piece-wise. Streaming SAX-inspired event-based API is included as well.
+`stream-json` is a micro-library of node.js stream components with minimal dependencies for creating custom data processors oriented on processing huge JSON files while requiring a minimal memory footprint. It can parse JSON files far exceeding available memory. Even individual primitive data items (keys, strings, and numbers) can be streamed piece-wise. Streaming SAX-inspired event-based API is included as well.
 
 Available components:
 
-* Streaming JSON parsers:
-  * Streaming JSON `Parser` is manually implemented based on `RegExp`.
-  * Streaming JSON `AltParser` implemented manually without regular expressions.
-  * Streaming JSON `ClassicParser` based on [parser-toolkit](http://github.com/uhop/parser-toolkit).
-* `Streamer`, which converts tokens into SAX-like event stream.
-* `Packer`, which can assemble numbers, strings, and object keys from individual chunks. It is useful, when user knows that individual data items can fit the available memory. Overall, it makes the API simpler.
-* `Combo`, which actually packs `Parser`, `Streamer`, and `Packer` together. **Its advantage over individual components is speed**.
-* `Filter`, which is a flexible tool to select only important sub-objects using either a regular expression, or a function.
-* `Emitter`, which converts an event stream into events by bridging `stream.Writable` with `EventEmitter`.
-* `Source`, which is a helper that connects streams using `pipe()` and converts an event stream on the end of pipe into events, similar to `Emitter`.
-* Various utilities:
-  * `Assembler` to assemble full objects from an event stream.
-  * `Stringer` to convert an event stream back to a JSON text stream.
-  * `StreamArray` handles a frequent use case: a huge array of relatively small objects similar to [Django](https://www.djangoproject.com/)-produced database dumps. It streams array components individually taking care of assembling them automatically.
-  * `StreamFilteredArray` is a companion for `StreamArray`. The difference is that it allows to filter out unneeded objects in an efficient way without assembling them fully.
-  * `FilterObjects` filters complete objects and primitives.
-  * `StreamObject` streams an object's key-value pairs individually taking care of assembling them automatically. Modeled after `StreamArray`.
-  * `StreamJsonObjects` supports [JSON Streaming](https://en.wikipedia.org/wiki/JSON_Streaming) protocol, where individual values are separated statically (like in `"{}[]"`), or with whitespaces (like in `"true 1 null"`). Modeled after `StreamArray`.
+* Streaming JSON [Parser](https://github.com/uhop/stream-json/wiki/Parser).
+  * It produces a SAX-like token stream.
+  * Optionally it can pack keys, strings, and numbers (controlled separately).
+  * The [main module](https://github.com/uhop/stream-json/wiki/Main-module) provides helpers to create a parser.
+* Filters to edit a token stream:
+  * [Pick](https://github.com/uhop/stream-json/wiki/Pick) selects desired objects.
+  * [Replace](https://github.com/uhop/stream-json/wiki/Replace) substitutes objects with a replacement.
+  * [Ignore](https://github.com/uhop/stream-json/wiki/Ignore) removes objects.
+  * [Filter](https://github.com/uhop/stream-json/wiki/Filter) filters tokens maintaining stream's validity.
+* Streamers to produce a stream of JavaScript objects.
+  * [StreamArray](https://github.com/uhop/stream-json/wiki/StreamArray) takes an array of objects and produces a stream of its components.
+    * It streams array components individually taking care of assembling them automatically.
+    * Created initially to deal with JSON files similar to [Django](https://www.djangoproject.com/)-produced database dumps.
+  * [StreamObject](https://github.com/uhop/stream-json/wiki/StreamObject) takes an object and produces a stream of its top-level properties.
+  * [StreamValues](https://github.com/uhop/stream-json/wiki/StreamValues) can handle a stream of JSON object.
+    * It supports [JSON Streaming](https://en.wikipedia.org/wiki/JSON_Streaming) protocol, where individual values are separated statically (like in `"{}[]"`), or with whitespaces (like in `"true 1 null"`).
+    * Useful to stream objects selected by `Pick`, or generated by other means.
+  * Essentials:
+    * [Assembler](https://github.com/uhop/stream-json/wiki/Assembler) interprets a token stream creating JavaScript objects.
+    * [Stringer](https://github.com/uhop/stream-json/wiki/Stringer) converts a token stream back into a JSON text stream.
+    * [Emitter](https://github.com/uhop/stream-json/wiki/Emitter) reads a token stream and emits each token as an event.
+      * It can greatly simplify data processing.
+  * Utilities:
+    * [emit()](https://github.com/uhop/stream-json/wiki/emit()) makes any stream component to emit tokens as events.
+    * [withParser()](https://github.com/uhop/stream-json/wiki/withParser()) helps to create stream components with a parser.
 
-Additionally a helper function is available in the main file, which creates a `Source` object with a default set of stream components.
+All components are meant to be building blocks to create flexible custom data processing pipelines. They can be extended and/or combined with custom code. They can be used together with [stream-chain](https://www.npmjs.com/package/stream-chain) to simplify data processing.
 
 This toolkit is distributed under New BSD license.
 
-See the full documentation below.
-
 ## Introduction
 
-The simplest example (streaming from a file):
-
 ```js
-var makeSource = require("stream-json");
-var source = makeSource();
+const {chain}  = require('stream-chain');
 
-var fs = require("fs");
+const {parser} = require('stream-json');
+const {pick}   = require('stream-json/filters/Pick');
+const {ignore} = require('stream-json/filters/Ignore');
+const {StreamValues} = require('stream-json/streamers/StreamValues');
 
-var objectCounter = 0;
-source.on("startObject", function(){ ++objectCounter; });
-source.on("end", function(){
-    console.log("Found ", objectCounter, " objects.");
-});
+const fs   = require('fs');
+const zlib = require('zlib');
 
-fs.createReadStream("sample.json").pipe(source.input);
+const pipeline = chain([
+  fs.createReadStream('sample.json.gz'),
+  zlib.createGunzip(),
+  parser(),
+  pick({filter: 'data'}),
+  ignore({filter: /\b_meta\b/i}),
+  streamValues(),
+  data => {
+    const value = data.value;
+    return value && value.department === 'accounting';
+  }
+]);
 
+let counter = 0;
+pipeline.on('data', () => ++counter);
+pipeline.on('end', () =>
+  console.log(`The accounting department has ${counter} employees.`));
 ```
 
+See the full documentation in [Wiki](https://github.com/uhop/stream-json/wiki).
+
 ## Installation
 
-```
+```bash
 npm install stream-json
+# or:
+yarn add stream-json
 ```
 
-## Documentation
+## Use
 
-### Parser
-
-This is the workhorse of the package. It is a [Transform](https://nodejs.org/api/stream.html#stream_class_stream_transform) stream, which consumes text, and produces a stream of tokens. It is always the first in a pipe chain being directly fed with a text from a file, a socket, the standard input, or any other text stream. Its `Writeable` part operates in a buffer mode, while its `Readable` part operates in an [objectMode](http://nodejs.org/api/stream.html#stream_object_mode).
-
-```js
-var Parser = require("stream-json/Parser");
-var parser = new Parser(options);
-
-// Example of use:
-var next = fs.createReadStream(fname).pipe(parser);
-```
-
-`options` can contain some technical parameters, and it rarely needs to be specified. You can find it thoroughly documented in [node.js' Stream documentation](http://nodejs.org/api/stream.html). Additionally it recognizes following properties:
-
-* `jsonStreaming` can be `true` or `false` (the default). If `true`, it can parse a stream of JSON objects as described in [JSON Streaming](https://en.wikipedia.org/wiki/JSON_Streaming) as "Concatenated JSON". Technically it will recognize "Line delimited JSON" as well.
-
-The test files for `Parser`: `tests/test_parser.js`, `tests\manual\test_parser.js`. Actually all test files in `tests/` use `Parser`.
-
-If you want to catch parsing errors, attach an error listener directly to a parser component — unlike data errors do not travel through stream pipes.
-
-### Streamer
-
-`Streamer` is a [Transform](https://nodejs.org/api/stream.html#stream_class_stream_transform) stream, which consumes a stream of tokens, and produces a stream of events. It is always the second in a pipe chain after the `Parser`. It knows JSON semantics and produces actionable events. It operates in an [objectMode](http://nodejs.org/api/stream.html#stream_object_mode).
-
-```js
-var Streamer = require("stream-json/Streamer");
-var streamer = new Streamer(options);
-
-// Example of use:
-var next = fs.createReadStream(fname).
-                pipe(parser).pipe(streamer);
-```
-
-`options` can contain some technical parameters, and it is rarely needs to be specified. You can find it thoroughly documented in [node.js' Stream documentation](http://nodejs.org/api/stream.html).
-
-Following is a list of all event objects produced by `Streamer`:
-
-```js
-{name: "startObject"};
-{name: "endObject"};
-
-{name: "startArray"};
-{name: "endArray"};
-
-{name: "startKey"};
-{name: "stringChunk", value: "actual string value"};
-{name: "endKey"};
-
-{name: "startString"};
-{name: "stringChunk", value: "actual string value"};
-{name: "endString"};
-
-{name: "startNumber"};
-{name: "numberChunk", value: "actual string value"};
-{name: "endNumber"};
-
-{name: "nullValue", value: null};
-{name: "trueValue", value: true};
-{name: "falseValue", value: false};
-
-```
-
-The event stream is well-formed:
-
-* All `startXXX` are balanced with `endXXX`.
-* Between `startKey` and `endKey` can be zero or more `stringChunk` events. No other event are allowed.
-* Between `startString` and `endString` can be zero or more `stringChunk` events. No other event are allowed.
-* Between `startNumber` and `endNumber` can be one or more `numberChunk` events. No other event are allowed.
-  * All number chunks combined constitute a valid number value.
-  * Number chunk values are strings, not numbers!
-* After `startObject` optional key-value pairs emitted in a strict pattern: a key-related events, a value, and this cycle can be continued until all key-value pairs are streamed.
-
-The test files for `Streamer`: `tests/test_streamer.js` and `tests/manual/test_streamer.js`.
-
-### Packer
-
-`Packer` is a [Transform](https://nodejs.org/api/stream.html#stream_class_stream_transform) stream, which passes through a stream of events, optionally assembles keys, strings, and/or numbers from chunks, and adds new events with assembled values. It is a companion  for `Streamer`, which frees users from implementing the assembling logic, when it is known that keys, strings, and/or numbers will fit in the available memory. It operates in an [objectMode](http://nodejs.org/api/stream.html#stream_object_mode).
-
-```js
-var Packer = require("stream-json/Packer");
-var packer = new Packer(options);
-
-// Example of use:
-var next = fs.createReadStream(fname).
-                pipe(parser).pipe(streamer).pipe(packer);
-```
-
-`options` contains some important parameters, and should be specified. It can contain some technical properties thoroughly documented in [node.js' Stream documentation](http://nodejs.org/api/stream.html). Additionally it recognizes following properties:
-
-* `packKeys` can be `true` or `false` (the default). If `true`, a key value is returned as a new event:
-
-  ```js
-  {name: "keyValue", value: "assembled key value"}
-  ```
-
-  `keyValue` event always follows `endKey`.
-* `packStrings` can be `true` or `false` (the default). If `true`, a string value is returned as a new event:
-
-  ```js
-  {name: "stringValue", value: "assembled string value"}
-  ```
-
-  `stringValue` event always follows `endString`.
-* `packNumbers` can be `true` or `false` (the default). If `true`, a number value is returned as a new event:
-
-  ```js
-  {name: "numberValue", value: "assembled number value"}
-  ```
-
-  `numberValue` event always follows `endNumber`.
-  `value` of this event is a string, not a number. If user wants to convert it to a number, they can do it themselves. The simplest way to do it (assuming your platform and JavaScript can handle it), is to force it to a number:
-
-  ```js
-  var n = +event.value;
-  ```
-
-The test files for `Packer`: `tests/test_packer.js` and `tests/manual/test_packer.js`.
-
-### Combo
-
-`Combo` is a [Transform](https://nodejs.org/api/stream.html#stream_class_stream_transform) stream, which combines `Parser`, `Streamer`, and `Packer`. It accepts the same extra options as `Parser` and `Packer`, and produces a stream of objects expected from `Streamer`, and augmented by `Packer`. Please refer to the documentation of those three components for more details.
-
-While logically `Combo` is a combination of three existing components, it has an important advantage: speed. The codes for `Parser`, `Streamer`, and `Packer` are merged together in one component avoiding overhead of node.js streams completely, which is significant. It is recommended to use `Combo` over a chain of `Parser` + `Streamer`, or `Parser` + `Streamer` + `Packer`.
-
-The test file for `Combo`: `tests/test_combo.js`.
-
-### Emitter
-
-`Emitter` is a [Writeable](https://nodejs.org/api/stream.html#stream_class_stream_writable) stream, which consumes a stream of events, and emits them on itself (all streams are instances of [EventEmitter](https://nodejs.org/api/events.html#events_class_events_eventemitter)). The standard `finish` event is used to indicate the end of a stream. It operates in an [objectMode](http://nodejs.org/api/stream.html#stream_object_mode).
-
-```js
-var Emitter = require("stream-json/Emitter");
-var emitter = new Emitter(options);
-
-// Example of use:
-
-emitter.on("startArray", function(){
-    console.log("array!");
-});
-emitter.on("numberValue", function(value){
-    console.log("number:", value);
-});
-emitter.on("finish", function(){
-    console.log("done");
-});
-
-fs.createReadStream(fname).
-    pipe(parser).pipe(streamer).pipe(packer).pipe(emitter);
-```
-
-`options` can contain some technical parameters, and it is rarely needs to be specified. You can find it thoroughly documented in [node.js' Stream documentation](http://nodejs.org/api/stream.html).
-
-The test file for `Emitter`: `tests/test_emitter.js`.
-
-### Filter
-
-`Filter` is a [Transform](https://nodejs.org/api/stream.html#stream_class_stream_transform) stream, which is an advance selector for sub-objects from a stream of events. It operates in an [objectMode](http://nodejs.org/api/stream.html#stream_object_mode).
-
-```js
-var Filter = require("stream-json/Filter");
-var filter = new Filter(options);
-
-// Example of use:
-var next = fs.createReadStream(fname).
-                pipe(parser).pipe(streamer).pipe(filter);
-```
-
-`options` contains some important parameters, and should be specified. It can contain some technical properties thoroughly documented in [node.js' Stream documentation](http://nodejs.org/api/stream.html). Additionally it recognizes following properties:
-
-* `separator` is a string to use to separate key and index values forming a path in a current object. By default it is `.` (a dot).
-* `filter` can be a regular expression, or a function. By default it allows all events.
-  * If it is a function, this function is called in a context of a `Filter` object with two parameters:
-    * `path`, which is an array of current key and index values. All keys are represented as strings, while all array indices are represented as numbers. It can be used to understand what kind of object we are dealing with.
-    * `event` is an event object described above.
-  The function should return a Boolean value, with `true` indicating that we are interested in this event, and it should be passed through.
-  * If it is a regular expression, then a current `path` is joined be a `separator` and tested against the regular expression. If a match was found, it indicates that the event should be passed through. Otherwise it will be rejected.
-* `defaultValue` is an array of event objects described above, which substitute skipped array items. By default it is `[{name: "nullValue", value: null}]` — a `null` value.
-  * `filter` can skip some array items. If it happens, `defaultValue` events are going to be inserted for every previously skipped item.
-    * If all array items were skipped, an empty array will be issued.
-    * Skipped items at the end of array are not substituted with `defaultValue`. A truncated array will be issued.
-  * **Important:** make sure that the sequence of events make sense, and do not violate JSON invariants. For example, all `startXXX` and `endXXX` should be properly balanced.
-  * It is possible to specify an empty array of events.
-
-`Filter` produces a well-formed event stream.
-
-The test files for `Filter`: `tests/test_filter.js` and `tests/manual/test_filter.js`.
-
-#### Path examples
-
-Given a JSON object:
-
-```js
-{"a": [true, false, 0, null]}
-```
-
-The path of `false` as an array:
-
-```js
-["a", 1]
-```
-
-The same path converted to a string joined by a default separator `.`:
-
-```js
-"a.1"
-```
-
-The top element of a stack can be `true` or `false`. The former means that an object was open, but no keys were processed. The latter means that an array was open, but no items were processed. Both values can be present in an array path, but not in a string path.
-
-### Source
-
-`Source` is a convenience object. It connects individual streams with pipes, and attaches itself to the end emitting all events on itself (just like `Emitter`). The standard `end` event is used to indicate the end of a stream. It is based on [EventEmitter](https://nodejs.org/api/events.html#events_class_events_eventemitter).
-
-```js
-var Source = require("stream-json/Source");
-var source = new Source([parser, streamer, packer]);
-
-// Example of use:
-
-source.on("startArray", function(){
-    console.log("array!");
-});
-source.on("numberValue", function(value){
-    console.log("number:", value);
-});
-
-fs.createReadStream(fname).pipe(source.input);
-```
-
-The constructor of `Source` accepts one mandatory parameter:
-
-* `streams` should be a non-empty array of pipeable streams. At the end the last stream should produce a stream of events.
-
-`Source` exposes three public properties:
-
-* `streams` — an array of streams so you can inspect them individually, if needed. They are connected sequentially in the array order.
-* `input` — the beginning of a pipeline, which should be used as an input for a JSON stream.
-* `output` — the end of a pipeline, which can be used to pipe the resulting stream of objects for further processing.
-
-The test files for `Source`: `tests/test_source.js` and `tests/manual/test_source.js`.
-
-### main: makeSource()
-
-The main file contains a helper function, which creates a commonly used configuration of streams, and returns a `Source` object.
-
-```js
-var makeSource = require("stream-json");
-var source = makeSource(options);
-
-// Example of use:
-
-source.on("startArray", function(){
-    console.log("array!");
-});
-source.on("numberValue", function(value){
-    console.log("number:", value);
-});
-
-fs.createReadStream(fname).pipe(source.input);
-```
-
-`options` can contain some technical parameters, and it is completely optional. You can find it thoroughly documented in [node.js' Stream documentation](http://nodejs.org/api/stream.html), and here. It is directly passed to `Combo`, so it will use its custom parameters.
-
-Algorithm:
-
-1. `makeSource()` checks if either of `packKeys`, `packStrings`, or `packNumbers` are specified in options.
-  1. If any of them are `true`, a `Combo` instance is created with `options`.
-  2. If all of them are unspecified, all pack flags are assumed to be `true`, and a `Combo` is created.
-2. A newly created instance of `Combo` is used to create a `Source` instance.
-
-The most common use case is to call `makeSource()` without parametrs. This scenario assumes that all key, string, and/or number values can be kept in memory, so user can use simplified events `keyValue`, `stringValue`, and `numberValue`.
-
-The test files for `makeSource()` are `tests/test_source.js`, `tests/manual/test_main.js`, and `tests/manual/test_chunk.js`.
-
-### ClassicParser
-
-It is a drop-in replacement for `Parser`, but it can emit whitespace, and token position information, yet it is slower than the main parser. It was the main parser for 0.1.x versions.
-
-It doesn't support `jsonStreaming` option.
-
-The test file for `ClassicParser`: `tests/test_classic.js`.
-
-### AltParser
-
-It is another drop-in replacement for `Parser`. Just like `ClassicParser` it can emit whitespace, and token position information, but can be slower than the current main parser on platforms with optimized `RegExp` implementation. It is faster than `Parser` on node.js 0.10.
-
-In general, its speed depends heavily on the implementation of regular expressions by node.js. When node.js has switched from an interpreted regular expressions, to the JIT compiled ones, both `ClassicParser`, and `Parser` got a nice performance boost. Yet, even the latest (as of 0.12) JIT compiler uses a simple yet non-linear algorithm to implement regular expressions instead of [NFA](http://en.wikipedia.org/wiki/Nondeterministic_finite_automaton) and/or [DFA](http://en.wikipedia.org/wiki/Deterministic_finite_automaton). Future enhancements to node.js would make `RegExp`-based parsers faster, potentially overtaking manually written JavaScript-only implementations.
-
-It doesn't support `jsonStreaming` option.
-
-The test file for `AltParser`: `tests/test_alternative.js`.
-
-### utils/Assembler
-
-A helper class to convert a JSON stream to a fully assembled JS object. It can be used to assemble sub-objects.
-
-```js
-var makeSource = require("stream-json");
-var Assembler  = require("stream-json/utils/Assembler");
-
-var source    = makeSource(options),
-    assembler = new Assembler();
-
-// Example of use:
-
-source.output.on("data", function(chunk){
-  assembler[chunk.name] && assembler[chunk.name](chunk.value);
-});
-source.output.on("end", function(){
-  // here is our fully assembled object:
-  console.log(assembler.current);
-});
-
-fs.createReadStream(fname).pipe(source.input);
-```
-
-`Assembler` is a simple state machine with an explicit stack. It exposes three properties:
-
-* `current` — an object we are working with at the moment. It can be either an object or an array.
-  * Initial value is `null`.
-  * If top-level object is a primitive value (`null`, `true`, `false`, a number, or a string), it will be placed in `current` too.
-* `key` — is a key value (a string) for a currently processed value, or `null`, if not expected.
-  * If `current` is an object, a primitive value will be added directly to it using a current value of `key`.
-    * After use `key` is assigned `null` to prevent memory leaks.
-  * If `current` is an array, a primitive value will be added directly to it by `push()`.
-* `stack` — an array of parent objects.
-  * `stack` always grows/shrinks by two items: a value of `current` and a value of `key`.
-  * When an object or an array is closed, it is added to its parent, which is removed from the stack to become a current object again.
-  * While adding to a parent a saved key is used if needed. Otherwise the second value is ignored.
-  * When an object or an array is started, the `current` object and `key` are saved to `stack`.
-
-Obviously `Assembler` should be used only when you are sure that the result will fit into memory. It automatically means that all primitive values (strings or numbers) are small enough to fit in memory too. As such `Assembler` is meant to be used after `Packer`, which reconstructs keys, strings, and numbers from possible chunks.
-
-On the other hand, we use `stream-json` when JSON streams are big, and `JSON.parse()` is not an option. But we use `Assembler` to assemble sub-objects. One way to do it is to start directing calls to `Assembler` when we already selected a sub-object with `Filter`. Another way is shown in `StreamArray`.
-
-The test file for `Assembler`: `tests/test_assembler.js`.
-
-### utils/Stringer
-
-This stream component converts an event stream (described in this document) back to a JSON text stream. The common use is to filter/transform an input stream, then pipe it back to the original text form.
-
-Its `Writeable` part operates in an [objectMode](http://nodejs.org/api/stream.html#stream_object_mode), while its `Readable` part operates in a buffer mode.
-
-```js
-var makeSource = require("stream-json");
-var Filter     = require("stream-json/Filter");
-var Stringer   = require("stream-json/utils/Stringer");
-
-var source    = makeSource(sourceOptions),
-    filter    = new Filter(filterOptions),
-    stringer  = new Stringer();
-
-// Example of use:
-source.output.pipe(filter).pipe(stringer).pipe(fs.createWriteStream(oname));
-fs.createReadStream(fname).pipe(source.input);
-```
-
-The test file for `Stringer`: `tests/test_stringer.js`.
-
-### utils/StreamArray
-
-This utility deals with a frequent use case: our JSON is an array of various sub-objects. The assumption is that while individual array items fit in memory, the array itself does not. Such files are frequently produced by various database dump utilities, e.g., [Django](https://www.djangoproject.com/)'s [dumpdata](https://docs.djangoproject.com/en/1.8/ref/django-admin/#dumpdata-app-label-app-label-app-label-model).
-
-It is a [Transform](https://nodejs.org/api/stream.html#stream_class_stream_transform) stream, which operates in an [objectMode](http://nodejs.org/api/stream.html#stream_object_mode).
-
-`StreamArray` produces a stream of objects in following format:
-
-```js
-{index, value}
-```
-
-Where `index` is a numeric index in the array starting from 0, and `value` is a corresponding value. All objects are produced strictly sequentially.
-
-```js
-var StreamArray = require("stream-json/utils/StreamArray");
-var stream = StreamArray.make();
-
-// Example of use:
-
-stream.output.on("data", function(object){
-  console.log(object.index, object.value);
-});
-stream.output.on("end", function(){
-  console.log("done");
-});
-
-fs.createReadStream(fname).pipe(stream.input);
-```
-
-`StreamArray` is a constructor, which optionally takes one object: `options`. `options` can contain some technical parameters, and it is rarely needs to be specified. You can find it thoroughly documented in [node.js' Stream documentation](http://nodejs.org/api/stream.html).
-
-Directly on `StreamArray` there is a class-level helper function `make()`, which helps to construct a proper pipeline. It is similar to `makeSource()` and takes the same argument `options`. Internally it creates and connects `Combo` and `StreamArray`, and returns an object with three properties:
-
-* `streams` — an array of streams so you can inspect them individually, if needed. They are connected sequentially in the array order.
-* `input` — the beginning of a pipeline, which should be used as an input for a JSON stream.
-* `output` — the end of a pipeline, which can be used for events, or to pipe the resulting stream of objects for further processing.
-
-The test file for `StreamArray`: `tests/test_array.js`.
-
-### utils/StreamObject
-
-Similar to `StreamArray`, except that instead of breaking an array into its elements it breaks an object into key/value pairs. Each pair has two properties: `key` and `value`.
-
-Like `StreamArray`, `StreamObject` is both a constructor and has a static `make()` function for common use cases.
-
-```js
-var StreamObject = require("stream-json/utils/StreamObject");
-var stream = StreamObject.make();
-
-// Example of use:
-
-stream.output.on("data", function(object){
-  console.log(object.key, object.value);
-});
-stream.output.on("end", function(){
-  console.log("done");
-});
-
-fs.createReadStream(fname).pipe(stream.input);
-```
-
-See the `StreamArray` documentation for more information.
-
-### utils/StreamFilteredArray
-
-This utility handles the same use case as `StreamArray`, but in addition it allows to check the objects as they are being built to reject, or accept them. Rejected objects are not assembled, and filtered out.
-
-It is a [Transform](https://nodejs.org/api/stream.html#stream_class_stream_transform) stream, which operates in an [objectMode](http://nodejs.org/api/stream.html#stream_object_mode).
-
-Just like `StreamArray`, `StreamFilteredArray` produces a stream of objects in following format:
-
-```js
-{index, value}
-```
-
-Where `index` is a numeric index in the array starting from 0, and `value` is a corresponding value. All objects are produced strictly sequentially.
-
-```js
-var StreamFilteredArray = require("stream-json/utils/StreamFilteredArray");
-
-function f(assembler){
-  // test only top-level objects in the array:
-  if(assembler.stack.length == 2 && assembler.key === null){
-    // make a decision depending on a boolean property "active":
-    if(assembler.current.hasOwnProperty("active")){
-      // "true" to accept, "false" to reject
-      return assembler.current.active;
-    }
-  }
-  // return undefined to indicate our uncertainty at this moment
-}
-
-var stream = StreamFilteredArray.make({objectFilter: f});
-
-// Example of use:
-
-stream.output.on("data", function(object){
-  console.log(object.index, object.value);
-});
-stream.output.on("end", function(){
-  console.log("done");
-});
-
-fs.createReadStream(fname).pipe(stream.input);
-```
-
-`StreamFilteredArray` is a constructor, which optionally takes one object: `options`. `options` can contain some technical parameters, which are rarely needs to be specified. You can find it thoroughly documented in [node.js' Stream documentation](http://nodejs.org/api/stream.html). But additionally it recognizes the following property:
-
-* `objectFilter` is a function, which takes an `Assembler` instance as its only argument, and may return following values to indicate its decision:
-  * any truthy value indicates that we are interested in this object. `StreamFilteredArray` will stop polling our filter function and will assemble the object for future use.
-  * `false` (the exact value) indicates that we should skip this object. `StreamFilteredArray` will stop polling our filter function, and will stop assembling the object, discarding it completely.
-  * any other falsy value indicates that we have not enough information (most likely because the object was not assembled yet to make a decision). `StreamFilteredArray` will poll our filter function next time the object changes.
-
-The default for `objectFilter` allows passing all objects.
-
-In general `objectFilter` is called on incomplete objects. It means that if a decision is based on a value of a certain properties, those properties could be unprocessed at that moment. In such case it is reasonable to delay a decision by returning a falsy (but not `false`) value, like `undefined`.
-
-Complete objects are not submitted to a filter function and accepted automatically. It means that all primitive values: booleans, numbers, strings, `null` objects are streamed, and not consulted with `objectFilter`.
-
-If you want to filter out complete objects, including primitive values, use `FilterObjects`.
-
-`StreamFilteredArray` instances expose one property:
-
-* `objectFilter` is a function, which us called for every top-level streamable object. It can be replaced with another function at any time. Usually it is replaced between objects after an accept/reject decision is made.
-
-Directly on `StreamFilteredArray` there is a class-level helper function `make()`, which is an exact clone of `StreamArray.make()`.
-
-The test file for `StreamFilteredArray`: `tests/test_filtered_array.js`.
-
-### utils/StreamJsonObjects
-
-This utility deals with [JSON Streaming](https://en.wikipedia.org/wiki/JSON_Streaming) -- "Concatenated JSON", when values are sent separated syntactically (e.g., `"{}[]"`) or with whitespaces (e.g., `"true 1 null"`), and "Line delimited JSON". The assumption is that while individual values fit in memory, the stream itself does not.
-
-This helper is modeled after `utils/StreamArray`.
-
-It is a [Transform](https://nodejs.org/api/stream.html#stream_class_stream_transform) stream, which operates in [objectMode](http://nodejs.org/api/stream.html#stream_object_mode).
-
-`StreamJsonObjects` produces a stream of objects in following format:
-
-```js
-{index, value}
-```
-
-Where `index` is a numeric (artificial) index in the stream starting from 0, and `value` is a corresponding value. All objects are produced strictly sequentially.
-
-```js
-var StreamJsonObjects = require("stream-json/utils/StreamJsonObjects");
-var stream = StreamJsonObjects.make();
-
-// Example of use:
-
-stream.output.on("data", function(object){
-  console.log(object.index, object.value);
-});
-stream.output.on("end", function(){
-  console.log("done");
-});
-
-fs.createReadStream(fname).pipe(stream.input);
-```
-
-`StreamJsonObjects` is a constructor, which optionally takes one object: `options`. `options` can contain some technical parameters, and it is rarely needs to be specified. You can find it thoroughly documented in [node.js' Stream documentation](http://nodejs.org/api/stream.html).
-
-Directly on `StreamJsonObjects` there is a class-level helper function `make()`, which helps to construct a proper pipeline. It is similar to `makeSource()` and takes the same argument `options`. Internally it creates and connects `Combo` and `StreamArray`, and returns an object with three properties:
-
-* `streams` — an array of streams so you can inspect them individually, if needed. They are connected sequentially in the array order.
-* `input` — the beginning of a pipeline, which should be used as an input for a JSON stream.
-* `output` — the end of a pipeline, which can be used for events, or to pipe the resulting stream of objects for further processing.
-
-The test file for `StreamJsonObjects`: `tests/test_json_stream.js`.
-
-### utils/FilterObjects
-
-This utility filters out complete objects (and primitive values) working with a stream in the same format as `StreamArray` and `StreamFilteredArray`:
-
-```js
-{index, value}
-```
-
-Where `index` is a numeric index in the array starting from 0, and `value` is a corresponding value. All objects are produced strictly sequentially.
-
-It is a [Transform](https://nodejs.org/api/stream.html#stream_class_stream_transform) stream, which operates in an [objectMode](http://nodejs.org/api/stream.html#stream_object_mode).
-
-```js
-var StreamArray   = require("stream-json/utils/StreamArray");
-var FilterObjects = require("stream-json/utils/FilterObjects");
-
-function f(item){
-  // accept all odd-indexed items, which are:
-  // true objects, but not arrays, or nulls
-  if(item.index % 2 && item.value &&
-      typeof item.value == "object" &&
-      !(item.value instanceof Array)){
-    return true;
-  }
-  return false;
-}
-
-var stream = StreamArray.make(),
-    filter = new FilterObjects({itemFilter: f});
-
-// Example of use:
-
-filter.on("data", function(object){
-  console.log(object.index, object.value);
-});
-filter.on("end", function(){
-  console.log("done");
-});
-
-fs.createReadStream(fname).pipe(stream.input).pipe(filter);
-```
-
-`FilterObjects` is a constructor, which optionally takes one object: `options`. `options` can contain some technical parameters, which are rarely needs to be specified. You can find it thoroughly documented in [node.js' Stream documentation](http://nodejs.org/api/stream.html). But additionally it recognizes the following property:
-
-* `itemFilter` is a function, which takes a `{index, value}` object as its only argument, and may return following values to indicate its decision:
-  * any truthy value to accept the object.
-  * any falsy value to reject the object.
-
-The default for `itemFilter` accepts all objects.
-
-`FilterObjects` instances expose one property:
-
-* `itemFilter` is a function, which us called for every top-level streamable object. It can be replaced with another function at any time.
-
-The test file for `FilterObjects`: `tests/test_filter_objects.js`.
-
-## Advanced use
-
 The whole library is organized as a set of small components, which can be combined to produce the most effective pipeline. All components are based on node.js [streams](http://nodejs.org/api/stream.html), and [events](http://nodejs.org/api/events.html). They implement all required standard APIs. It is easy to add your own components to solve your unique tasks.
 
-The code of all components are compact and simple. Please take a look at their source code to see how things are implemented, so you can produce your own components in no time.
+The code of all components is compact and simple. Please take a look at their source code to see how things are implemented, so you can produce your own components in no time.
 
 Obviously, if a bug is found, or a way to simplify existing components, or new generic components are created, which can be reused in a variety of projects, don't hesitate to open a ticket, and/or create a pull request.
 
-## FAQ
-
-### What if my utf-8 data is decoded incorrectly?
-
-`stream-json` does not decode utf-8 relying on Node to do it correctly. Apparently in some cases Node can fail to decode multi-byte characters correctly, when they are split between different buffers. If you encounter that problem (I did not see it in the wild yet), you can solve it by piping an input stream through a sanitizer before sending it to `stream-json` parser. These two packages look promising, and appear to be doing the right thing:
-* https://www.npmjs.com/package/utf8-stream
-* https://www.npmjs.com/package/utf8-align-stream
-
-## Credits
-
-The test file `tests/sample.json.gz` is a combination of several publicly available datasets merged and compressed with gzip:
-
-* a snapshot of publicly available [Japanese statistics on birth and marriage in JSON](http://dataforjapan.org/dataset/birth-stat/resource/42799d3c-ecee-4b35-9f5a-7fec30596aa2).
-* a snapshot of publicly available [US Department of Housing and Urban Development - HUD's published metadata catalog (Schema Version 1.1)](https://catalog.data.gov/dataset/data-catalog).
-* a small fake sample made up by me featuring non-ASCII keys, non-ASCII strings, and primitive data missing in other two samples.
-
-## Apendix A: tokens
-
-`Parser`, `AltParser`, and `ClassicParser` produce a stream of tokens cortesy of [parser-toolkit](http://github.com/uhop/parser-toolkit). While normally user should use `Streamer` to convert them to a much simpler JSON-aware event stream, or use `Combo` directly, in some cases it can be advantageous to deal with raw tokens.
-
-Each token is an object with following properties:
-
-* `id` is a string, which uniquely identifies a token.
-* `value` is a string, which corresponds to this token, and was actually matched.
-* `line` is a line number, where this token was found. All lines are counted from 1.
-* `pos` is a position number inside a line (in characters, so `\t` is one character). Position is counted from 1.
-
-Warning: `Parser` does not incliude `line` and `pos` in its tokens.
-
-JSON grammar is defined in `Grammar.js`. It is taken almost verbatim from [JSON.org](http://json.org/).
-
-Following tokens are produced (listed by `id`):
-
-* `ws`: white spaces, usually ignored. (Not produced by `Parser`.)
-* `-`: a unary negation used in a negative number either to start a number, or as an exponent sign.
-* `+`: used as an exponent sign.
-* `0`: zero, as is - '0'.
-* `nonZero`: non-zero digit - `/[1-9]/`.
-* `.`: a decimal point used in a number.
-* `exponent`: 'e' or 'E' as an exponent symbol in a number written in scientific notation.
-* `numericChunk`: a string of digits.
-* `"`: a double quote, used to open and close a string.
-* `plainChunk`: a string of non-escaped characters, used inside a string.
-* `escapedChars`: an escaped character, used inside a string.
-* `true`: represents a literal `true`.
-* `false`: represents a literal `false`.
-* `null`: represents a literal `null`.
-* `{`: starts an object literal.
-* `}`: closes an object literal.
-* `[`: starts an array literal.
-* `]`: closes an array literal.
-* `,`: separates components of an array, or an object.
-* `:`: separates a key and its value in an object literal.
-
 ## Release History
 
+- 1.0.0 *the first 1.0 release.*
 - 0.6.1 *the technical release.*
@@ -719,0 +98,0 @@ - 0.6.0 *added Stringer to convert event streams back to JSON.*

tests/Counter.js

@@ -1,49 +0,50 @@
-"use strict";
+'use strict';
 
-
-function Counter(){
-	this.objects = 0;
-	this.keys    = 0;
-	this.arrays  = 0;
-	this.nulls   = 0;
-	this.trues   = 0;
-	this.falses  = 0;
-	this.numbers = 0;
-	this.strings = 0;
+function Counter() {
+  this.objects = 0;
+  this.keys = 0;
+  this.arrays = 0;
+  this.nulls = 0;
+  this.trues = 0;
+  this.falses = 0;
+  this.numbers = 0;
+  this.strings = 0;
 }
 
-Counter.walk = function walk(o, counter){
-	switch(typeof o){
-		case "string":
-			++counter.strings;
-			return;
-		case "number":
-			++counter.numbers;
-			return;
-		case "boolean":
-			if(o){
-				++counter.trues;
-			}else{
-				++counter.falses;
-			}
-			return;
-	}
-	if(o === null){
-		++counter.nulls;
-		return;
-	}
-	if(o instanceof Array){
-		++counter.arrays;
-		o.forEach(function(o){ walk(o, counter); });
-		return;
-	}
-	++counter.objects;
-	for(var key in o){
-		if(o.hasOwnProperty(key)){
-			++counter.keys;
-			walk(o[key], counter);
-		}
-	}
+Counter.walk = function walk(o, counter) {
+  switch (typeof o) {
+    case 'string':
+      ++counter.strings;
+      return;
+    case 'number':
+      ++counter.numbers;
+      return;
+    case 'boolean':
+      if (o) {
+        ++counter.trues;
+      } else {
+        ++counter.falses;
+      }
+      return;
+  }
+  if (o === null) {
+    ++counter.nulls;
+    return;
+  }
+  if (o instanceof Array) {
+    ++counter.arrays;
+    o.forEach(function(o) {
+      walk(o, counter);
+    });
+    return;
+  }
+  ++counter.objects;
+  for (let key in o) {
+    if (o.hasOwnProperty(key)) {
+      ++counter.keys;
+      walk(o[key], counter);
+    }
+  }
 };
 
 module.exports = Counter;

tests/ReadString.js

@@ -1,24 +0,26 @@
-var util = require("util");
-var Readable = require("stream").Readable;
+const {Readable} = require('stream');
 
+class ReadString extends Readable {
+  static readString(string, quant, options) {
+    return new ReadString(string, quant, options);
+  }
 
-function ReadString(string, quant, options){
-	Readable.call(this, options);
-	this._string = string;
-	this._quant  = quant;
+  constructor(string, quant, options) {
+    super(options);
+    this._string = string;
+    this._quant = quant;
+  }
+  _read(size) {
+    if (isNaN(this._quant)) {
+      this.push(this._string, 'utf8');
+    } else {
+      for (let i = 0; i < this._string.length; i += this._quant) {
+        this.push(this._string.substr(i, this._quant), 'utf8');
+      }
+    }
+    this.push(null);
+  }
 }
-util.inherits(ReadString, Readable);
+ReadString.make = ReadString.readString;
 
-ReadString.prototype._read = function read(size){
-	if(isNaN(this._quant)){
-		this.push(this._string, "utf8");
-	}else{
-		for(var i = 0; i < this._string.length; i += this._quant){
-			this.push(this._string.substr(i, this._quant), "utf8");
-		}
-	}
-	this.push(null);
-};
-
-
 module.exports = ReadString;

tests/test_array.js

@@ -1,47 +0,147 @@
-"use strict";
+'use strict';
 
+const unit = require('heya-unit');
 
-var unit = require("heya-unit");
+const ReadString = require('./ReadString');
+const StreamArray = require('../src/streamers/StreamArray');
 
-var ReadString  = require("./ReadString");
-var StreamArray = require("../utils/StreamArray");
+unit.add(module, [
+  function test_array(t) {
+    const async = t.startAsync('test_array');
 
+    const stream = StreamArray.withParser(),
+      pattern = [0, 1, true, false, null, {}, [], {a: 'b'}, ['c']],
+      result = [];
 
-unit.add(module, [
-	function test_array_fail(t){
-		var async = t.startAsync("test_array_fail");
+    stream.output.on('data', object => (result[object.key] = object.value));
+    stream.output.on('end', () => {
+      eval(t.TEST('t.unify(pattern, result)'));
+      async.done();
+    });
 
-		var stream = StreamArray.make();
+    new ReadString(JSON.stringify(pattern)).pipe(stream.input);
+  },
+  function test_array_fail(t) {
+    const async = t.startAsync('test_array_fail');
 
-		stream.output.on("data", function(value){
-			eval(t.TEST("!'We shouldn\'t be here.'"));
-		});
-		stream.output.on("error", function(err){
-			eval(t.TEST("err"));
-			async.done();
-		});
-		stream.output.on("end", function(value){
-			eval(t.TEST("!'We shouldn\'t be here.'"));
-			async.done();
-		});
+    const stream = StreamArray.withParser();
 
-		new ReadString(" true ").pipe(stream.input);
-	},
-	function test_array(t){
-		var async = t.startAsync("test_array");
+    stream.on('data', value => eval(t.TEST("!'We shouldn't be here.'")));
+    stream.on('error', e => {
+      eval(t.TEST('e'));
+      async.done();
+    });
+    stream.on('end', value => {
+      eval(t.TEST("!'We shouldn't be here.'"));
+      async.done();
+    });
 
-		var stream  = StreamArray.make(),
-			pattern = [0, 1, true, false, null, {}, [], {a: "b"}, ["c"]],
-			result  = [];
+    new ReadString(' true ').pipe(stream);
+  },
+  function test_array_filter(t) {
+    const async = t.startAsync('test_array_filter');
 
-		stream.output.on("data", function(object){
-			result[object.index] = object.value;
-		});
-		stream.output.on("end", function(){
-			eval(t.TEST("t.unify(pattern, result)"));
-			async.done();
-		});
+    const f = assembler => {
+      if (assembler.depth == 2 && assembler.key === null) {
+        if (assembler.current instanceof Array) {
+          return false; // reject
+        }
+        switch (assembler.current.a) {
+          case 'accept':
+            return true; // accept
+          case 'reject':
+            return false; // reject
+        }
+      }
+      // undecided
+    };
 
-		new ReadString(JSON.stringify(pattern)).pipe(stream.input);
-	}
+    const stream = StreamArray.withParser({objectFilter: f}),
+      input = [
+        0,
+        1,
+        true,
+        false,
+        null,
+        {},
+        [],
+        {a: 'reject', b: [[[]]]},
+        ['c'],
+        {a: 'accept'},
+        {a: 'neutral'},
+        {x: true, a: 'reject'},
+        {y: null, a: 'accept'},
+        {z: 1234, a: 'neutral'},
+        {w: '12', a: 'neutral'}
+      ],
+      result = [];
+
+    stream.output.on('data', object => result.push(object.value));
+    stream.output.on('end', () => {
+      result.forEach(o => {
+        if (typeof o == 'object' && o) {
+          eval(t.TEST('!(o instanceof Array)'));
+          eval(t.TEST("o.a === 'accept'"));
+        } else {
+          eval(t.TEST('false')); // shouldn't be here
+        }
+      });
+      async.done();
+    });
+
+    new ReadString(JSON.stringify(input)).pipe(stream.input);
+  },
+  function test_array_filter_include(t) {
+    const async = t.startAsync('test_array_filter_include');
+
+    const f = assembler => {
+      if (assembler.depth == 2 && assembler.key === null) {
+        if (assembler.current instanceof Array) {
+          return false; // reject
+        }
+        switch (assembler.current.a) {
+          case 'accept':
+            return true; // accept
+          case 'reject':
+            return false; // reject
+        }
+      }
+      // undecided
+    };
+
+    const stream = StreamArray.withParser({objectFilter: f, includeUndecided: true}),
+      input = [
+        0,
+        1,
+        true,
+        false,
+        null,
+        {},
+        [],
+        {a: 'reject', b: [[[]]]},
+        ['c'],
+        {a: 'accept'},
+        {a: 'neutral'},
+        {x: true, a: 'reject'},
+        {y: null, a: 'accept'},
+        {z: 1234, a: 'neutral'},
+        {w: '12', a: 'neutral'}
+      ],
+      result = [];
+
+    stream.output.on('data', object => result.push(object.value));
+    stream.output.on('end', () => {
+      result.forEach(o => {
+        if (typeof o == 'object' && o) {
+          eval(t.TEST('!(o instanceof Array)'));
+          eval(t.TEST("o.a !== 'reject'"));
+        } else {
+          eval(t.TEST("o === null || typeof o != 'object'"));
+        }
+      });
+      async.done();
+    });
+
+    new ReadString(JSON.stringify(input)).pipe(stream.input);
+  }
 ]);

tests/test_assembler.js

@@ -1,41 +0,88 @@
-"use strict";
+'use strict';
 
+const unit = require('heya-unit');
 
-var unit = require("heya-unit");
+const fs = require('fs'),
+  path = require('path'),
+  zlib = require('zlib');
 
-var fs = require("fs"), path = require("path"), zlib = require("zlib");
+const ReadString = require('./ReadString');
+const makeParser = require('../src/main');
+const Assembler = require('../src/Assembler');
 
-var makeSource = require("../main");
-var Assembler  = require("../utils/Assembler");
+unit.add(module, [
+  function test_assembler(t) {
+    const async = t.startAsync('test_assembler');
 
+    let object = null;
+    const parser = makeParser(),
+      assembler = Assembler.connectTo(parser);
 
-unit.add(module, [
-	function test_escaped(t){
-		var async = t.startAsync("test_escaped");
+    parser.on('end', () => {
+      eval(t.TEST('t.unify(assembler.current, object)'));
+      async.done();
+    });
 
-		var source = makeSource(),
-			assembler = new Assembler(),
-			object = null;
+    fs.readFile(path.resolve(__dirname, './sample.json.gz'), (err, data) => {
+      if (err) {
+        throw err;
+      }
+      zlib.gunzip(data, (err, data) => {
+        if (err) {
+          throw err;
+        }
 
-		source.output.on("data", function(chunk){
-			assembler[chunk.name] && assembler[chunk.name](chunk.value);
-		});
-		source.output.on("end", function(){
-			eval(t.TEST("t.unify(assembler.current, object)"));
-			async.done();
-		});
+        object = JSON.parse(data.toString());
 
+        fs.createReadStream(path.resolve(__dirname, './sample.json.gz'))
+          .pipe(zlib.createGunzip())
+          .pipe(parser);
+      });
+    });
+  },
+  function test_assembler_no_streaming(t) {
+    const async = t.startAsync('test_assembler_no_streaming');
 
-		fs.readFile(path.resolve(__dirname, "./sample.json.gz"), function(err, data){
-			if(err){ throw err; }
-			zlib.gunzip(data, function(err, data){
-				if(err){ throw err; }
+    let object = null;
+    const parser = makeParser({streamValues: false}),
+      assembler = Assembler.connectTo(parser);
 
-				object = JSON.parse(data.toString());
+    parser.on('end', () => {
+      eval(t.TEST('t.unify(assembler.current, object)'));
+      async.done();
+    });
 
-				fs.createReadStream(path.resolve(__dirname, "./sample.json.gz")).
-					pipe(zlib.createGunzip()).pipe(source.input);
-			});
-		});
-	}
+    fs.readFile(path.resolve(__dirname, './sample.json.gz'), (err, data) => {
+      if (err) {
+        throw err;
+      }
+      zlib.gunzip(data, (err, data) => {
+        if (err) {
+          throw err;
+        }
+
+        object = JSON.parse(data.toString());
+
+        fs.createReadStream(path.resolve(__dirname, './sample.json.gz'))
+          .pipe(zlib.createGunzip())
+          .pipe(parser);
+      });
+    });
+  },
+  function test_assembler_json_objects(t) {
+    const async = t.startAsync('test_stringer_json_stream_primitives');
+
+    const parser = makeParser({jsonStreaming: true}),
+      assembler = Assembler.connectTo(parser),
+      pattern = [1, 2, "zzz", "z'z\"z", null, true, false, 1, [], null, {}, true, {"a": "b"}],
+      result = [];
+
+    assembler.on('done', asm => result.push(asm.current));
+    parser.on('end', () => {
+      eval(t.TEST('t.unify(result, pattern)'));
+      async.done();
+    });
+
+    new ReadString(pattern.map(value => JSON.stringify(value)).join(' ')).pipe(parser);
+  }
 ]);

tests/test_emitter.js

@@ -1,55 +0,56 @@
-"use strict";
+'use strict';
 
+const unit = require('heya-unit');
 
-var unit = require("heya-unit");
+const fs = require('fs'),
+  path = require('path'),
+  zlib = require('zlib');
 
-var fs = require("fs"), path = require("path"), zlib = require("zlib");
+const Parser = require('../src/Parser');
+const Emitter = require('../src/Emitter');
+const Counter = require('./Counter');
 
-var Parser   = require("../Parser");
-var Streamer = require("../Streamer");
-var Packer   = require("../Packer");
-var Emitter  = require("../Emitter");
-var Counter  = require("./Counter");
-
-
 unit.add(module, [
-	function test_emitter(t){
-		var async = t.startAsync("test_emitter");
+  function test_emitter(t) {
+    const async = t.startAsync('test_emitter');
 
-		var plainCounter   = new Counter(),
-			emitterCounter = new Counter(),
-			parser   = new Parser(),
-			streamer = new Streamer(),
-			packer   = new Packer({packKeys: true, packStrings: true, packNumbers: true}),
-			emitter  = new Emitter();
+    const plainCounter = new Counter(),
+      emitterCounter = new Counter(),
+      parser = new Parser(),
+      emitter = new Emitter();
 
-		parser.pipe(streamer).pipe(packer).pipe(emitter);
+    parser.pipe(emitter);
 
-		emitter.on("startObject", function(){ ++emitterCounter.objects; });
-		emitter.on("keyValue",    function(){ ++emitterCounter.keys; });
-		emitter.on("startArray",  function(){ ++emitterCounter.arrays; });
-		emitter.on("nullValue",   function(){ ++emitterCounter.nulls; });
-		emitter.on("trueValue",   function(){ ++emitterCounter.trues; });
-		emitter.on("falseValue",  function(){ ++emitterCounter.falses; });
-		emitter.on("numberValue", function(){ ++emitterCounter.numbers; });
-		emitter.on("stringValue", function(){ ++emitterCounter.strings; });
+    emitter.on('startObject', () => ++emitterCounter.objects);
+    emitter.on('keyValue', () => ++emitterCounter.keys);
+    emitter.on('startArray', () => ++emitterCounter.arrays);
+    emitter.on('nullValue', () => ++emitterCounter.nulls);
+    emitter.on('trueValue', () => ++emitterCounter.trues);
+    emitter.on('falseValue', () => ++emitterCounter.falses);
+    emitter.on('numberValue', () => ++emitterCounter.numbers);
+    emitter.on('stringValue', () => ++emitterCounter.strings);
 
-		emitter.on("finish", function(){
-			eval(t.TEST("t.unify(plainCounter, emitterCounter)"));
-			async.done();
-		});
+    emitter.on('finish', () => {
+      eval(t.TEST('t.unify(plainCounter, emitterCounter)'));
+      async.done();
+    });
 
-		fs.readFile(path.resolve(__dirname, "./sample.json.gz"), function(err, data){
-			if(err){ throw err; }
-			zlib.gunzip(data, function(err, data){
-				if(err){ throw err; }
+    fs.readFile(path.resolve(__dirname, './sample.json.gz'), (err, data) => {
+      if (err) {
+        throw err;
+      }
+      zlib.gunzip(data, (err, data) => {
+        if (err) {
+          throw err;
+        }
 
-				var o = JSON.parse(data);
-				Counter.walk(o, plainCounter);
+        const o = JSON.parse(data);
+        Counter.walk(o, plainCounter);
 
-				fs.createReadStream(path.resolve(__dirname, "./sample.json.gz")).
-					pipe(zlib.createGunzip()).pipe(parser);
-			});
-		});
-	}
+        fs.createReadStream(path.resolve(__dirname, './sample.json.gz'))
+          .pipe(zlib.createGunzip())
+          .pipe(parser);
+      });
+    });
+  }
 ]);

tests/test_escaped.js

@@ -1,36 +0,26 @@
-"use strict";
+'use strict';
 
+const unit = require('heya-unit');
 
-var unit = require("heya-unit");
+const Assembler = require('../src/Assembler');
+const Parser = require('../src/Parser');
+const ReadString = require('./ReadString');
 
-var Assembler = require("../utils/Assembler");
-
-var Parser   = require("../Parser");
-var Streamer = require("../Streamer");
-var Packer   = require("../Packer");
-
-var ReadString = require("./ReadString");
-
-
 unit.add(module, [
-	function test_escaped(t){
-		var async = t.startAsync("test_escaped");
+  function test_escaped(t) {
+    const async = t.startAsync('test_escaped');
 
-		var object = {
-				stringWithTabsAndNewlines: "Did it work?\nNo...\t\tI don't think so...",
-				anArray: [1, 2, true, "tabs?\t\t\t\u0001\u0002\u0003", false]
-			},
-			input = JSON.stringify(object),
-			pipeline = new ReadString(input).pipe(new Parser()).pipe(new Streamer()).
-				pipe(new Packer({packKeys: true, packStrings: true, packNumbers: true})),
-			assembler = new Assembler();
+    const object = {
+        stringWithTabsAndNewlines: "Did it work?\nNo...\t\tI don't think so...",
+        anArray: [1, 2, true, 'tabs?\t\t\t\u0001\u0002\u0003', false]
+      },
+      input = JSON.stringify(object),
+      pipeline = new ReadString(input).pipe(new Parser()),
+      assembler = Assembler.connectTo(pipeline);
 
-		pipeline.on("data", function(chunk){
-			assembler[chunk.name] && assembler[chunk.name](chunk.value);
-		});
-		pipeline.on("end", function(){
-			eval(t.TEST("t.unify(assembler.current, object)"));
-			async.done();
-		});
-	}
+    pipeline.on('end', () => {
+      eval(t.TEST('t.unify(assembler.current, object)'));
+      async.done();
+    });
+  }
 ]);

tests/test_filter.js

@@ -1,115 +0,119 @@
-"use strict";
+'use strict';
 
+const unit = require('heya-unit');
 
-var unit = require("heya-unit");
+const {chain} = require('stream-chain');
 
-var Parser   = require("../Parser");
-var Streamer = require("../Streamer");
-var Combo    = require("../Combo");
-var Filter   = require("../Filter");
+const {parser} = require('../src/Parser');
+const {filter} = require('../src/filters/Filter');
+const Assembler = require('../src/Assembler');
 
-var Asm = require("../utils/Assembler");
+const {readString} = require('./ReadString');
 
-var ReadString = require("./ReadString");
-
-
 unit.add(module, [
-	function test_filter(t){
-		var async = t.startAsync("test_filter");
+  function test_filter(t) {
+    const async = t.startAsync('test_filter');
 
-		var input = '{"a": 1, "b": true, "c": ["d"]}',
-			pipeline = new ReadString(input).pipe(new Parser()).pipe(new Streamer()).
-				pipe(new Filter({filter: /^(|a|c)$/})),
-			result = [];
+    const input = '{"a": 1, "b": true, "c": ["d"]}',
+      pipeline = chain([readString(input), parser({packKeys: true, packValues: false}), filter({filter: /^(|a|c)$/})]),
+      result = [];
 
-		pipeline.on("data", function(chunk){
-			result.push({name: chunk.name, val: chunk.value});
-		});
-		pipeline.on("end", function(){
-			eval(t.ASSERT("result.length === 15"));
-			eval(t.TEST("result[0].name === 'startObject'"));
-			eval(t.TEST("result[1].name === 'startKey'"));
-			eval(t.TEST("result[2].name === 'stringChunk' && result[2].val === 'a'"));
-			eval(t.TEST("result[3].name === 'endKey'"));
-			eval(t.TEST("result[4].name === 'keyValue' && result[4].val === 'a'"));
-			eval(t.TEST("result[5].name === 'startNumber'"));
-			eval(t.TEST("result[6].name === 'numberChunk' && result[6].val === '1'"));
-			eval(t.TEST("result[7].name === 'endNumber'"));
-			eval(t.TEST("result[8].name === 'startKey'"));
-			eval(t.TEST("result[9].name === 'stringChunk' && result[9].val === 'c'"));
-			eval(t.TEST("result[10].name === 'endKey'"));
-			eval(t.TEST("result[11].name === 'keyValue' && result[11].val === 'c'"));
-			eval(t.TEST("result[12].name === 'startArray'"));
-			eval(t.TEST("result[13].name === 'endArray'"));
-			eval(t.TEST("result[14].name === 'endObject'"));
-			async.done();
-		});
-	},
-	function test_filter_deep(t){
-		var async = t.startAsync("test_filter_deep");
+    pipeline.on('data', chunk => result.push({name: chunk.name, val: chunk.value}));
+    pipeline.on('end', () => {
+      eval(t.ASSERT('result.length === 15'));
+      eval(t.TEST("result[0].name === 'startObject'"));
+      eval(t.TEST("result[1].name === 'startKey'"));
+      eval(t.TEST("result[2].name === 'stringChunk' && result[2].val === 'a'"));
+      eval(t.TEST("result[3].name === 'endKey'"));
+      eval(t.TEST("result[4].name === 'keyValue' && result[4].val === 'a'"));
+      eval(t.TEST("result[5].name === 'startNumber'"));
+      eval(t.TEST("result[6].name === 'numberChunk' && result[6].val === '1'"));
+      eval(t.TEST("result[7].name === 'endNumber'"));
+      eval(t.TEST("result[8].name === 'startKey'"));
+      eval(t.TEST("result[9].name === 'stringChunk' && result[9].val === 'c'"));
+      eval(t.TEST("result[10].name === 'endKey'"));
+      eval(t.TEST("result[11].name === 'keyValue' && result[11].val === 'c'"));
+      eval(t.TEST("result[12].name === 'startArray'"));
+      eval(t.TEST("result[13].name === 'endArray'"));
+      eval(t.TEST("result[14].name === 'endObject'"));
+      async.done();
+    });
+  },
+  function test_filter_no_streaming(t) {
+    const async = t.startAsync('test_filter_no_streaming');
 
-		var data = {a: {b: {c: 1}}, b: {b: {c: 2}}, c: {b: {c: 3}}};
+    const input = '{"a": 1, "b": true, "c": ["d"]}',
+      pipeline = chain([readString(input), parser({packKeys: true, packValues: false, streamValues: false}), filter({filter: /^(|a|c)$/, streamValues: false})]),
+      result = [];
 
-		const pipeline = new ReadString(JSON.stringify(data)).
-		    pipe(new Combo({packKeys: true, packStrings: true, packNumbers: true})).
-		    pipe(new Filter({filter: /^(?:a|c)\.b\b/}));
+    pipeline.on('data', chunk => result.push({name: chunk.name, val: chunk.value}));
+    pipeline.on('end', () => {
+      eval(t.ASSERT('result.length === 9'));
+      eval(t.TEST("result[0].name === 'startObject'"));
+      eval(t.TEST("result[1].name === 'keyValue' && result[1].val === 'a'"));
+      eval(t.TEST("result[2].name === 'startNumber'"));
+      eval(t.TEST("result[3].name === 'numberChunk' && result[3].val === '1'"));
+      eval(t.TEST("result[4].name === 'endNumber'"));
+      eval(t.TEST("result[5].name === 'keyValue' && result[5].val === 'c'"));
+      eval(t.TEST("result[6].name === 'startArray'"));
+      eval(t.TEST("result[7].name === 'endArray'"));
+      eval(t.TEST("result[8].name === 'endObject'"));
+      async.done();
+    });
+  },
+  function test_filter_deep(t) {
+    const async = t.startAsync('test_filter_deep');
 
-		const asm = new Asm();
+    const data = {a: {b: {c: 1}}, b: {b: {c: 2}}, c: {b: {c: 3}}};
 
-		pipeline.on('data', function (chunk) {
-		    asm[chunk.name] && asm[chunk.name](chunk.value);
-		});
+    const pipeline = chain([readString(JSON.stringify(data)), parser({streamValues: false}), filter({filter: /^(?:a|c)\.b\b/})]);
 
-		pipeline.on('end', function () {
-		    eval(t.TEST("t.unify(asm.current, {a: {b: {c: 1}}, c: {b: {c: 3}}})"));
-			async.done();
-		});
+    const asm = Assembler.connectTo(pipeline);
 
-	},
-	function test_filter_array(t){
-		var async = t.startAsync("test_filter_array");
+    pipeline.on('end', () => {
+      eval(t.TEST('t.unify(asm.current, {a: {b: {c: 1}}, c: {b: {c: 3}}})'));
+      async.done();
+    });
+  },
+  function test_filter_array(t) {
+    const async = t.startAsync('test_filter_array');
 
-		var data = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10];
+    const data = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10];
 
-		const pipeline = new ReadString(JSON.stringify(data)).
-		    pipe(new Combo({packKeys: true, packStrings: true, packNumbers: true})).
-		    pipe(new Filter({filter: function (stack) {
-				return stack.length == 1 && typeof stack[0] == "number" && stack[0] % 2;
-			}}));
+    const pipeline = chain([
+      readString(JSON.stringify(data)),
+      parser(),
+      filter({
+        filter: stack => stack.length == 1 && typeof stack[0] == 'number' && stack[0] % 2
+      })
+    ]);
 
-		const asm = new Asm();
+    const asm = Assembler.connectTo(pipeline);
 
-		pipeline.on('data', function (chunk) {
-		    asm[chunk.name] && asm[chunk.name](chunk.value);
-		});
+    pipeline.on('end', () => {
+      eval(t.TEST('t.unify(asm.current, [2, 4, 6, 8, 10])'));
+      async.done();
+    });
+  },
+  function test_filter_default_value(t) {
+    const async = t.startAsync('test_filter_default_value');
 
-		pipeline.on('end', function () {
-		    eval(t.TEST("t.unify(asm.current, [null, 2, null, 4, null, 6, null, 8, null, 10])"));
-			async.done();
-		});
+    const data = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10];
 
-	},
-	function test_filter_default_value(t){
-		var async = t.startAsync("test_filter_default_value");
+    const pipeline = chain([
+      readString(JSON.stringify(data)),
+      parser(),
+      filter({
+        defaultValue: [],
+        filter: stack => stack.length == 1 && typeof stack[0] == 'number' && stack[0] % 2
+      })
+    ]);
 
-		var data = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10];
+    const asm = Assembler.connectTo(pipeline);
 
-		const pipeline = new ReadString(JSON.stringify(data)).
-		    pipe(new Combo({packKeys: true, packStrings: true, packNumbers: true})).
-		    pipe(new Filter({defaultValue: [], filter: function (stack) {
-				return stack.length == 1 && typeof stack[0] == "number" && stack[0] % 2;
-			}}));
-
-		const asm = new Asm();
-
-		pipeline.on('data', function (chunk) {
-		    asm[chunk.name] && asm[chunk.name](chunk.value);
-		});
-
-		pipeline.on('end', function () {
-		    eval(t.TEST("t.unify(asm.current, [2, 4, 6, 8, 10])"));
-			async.done();
-		});
-
-	}
+    pipeline.on('end', () => {
+      eval(t.TEST('t.unify(asm.current, [2, 4, 6, 8, 10])'));
+      async.done();
+    });
+  }
 ]);

tests/test_json_stream.js

@@ -1,52 +0,160 @@
-"use strict";
+'use strict';
 
+const unit = require('heya-unit');
 
-var unit = require("heya-unit");
+const StreamValues = require('../src/streamers/StreamValues');
 
-var ReadString  = require("./ReadString");
-var StreamJsonObjects = require("../utils/StreamJsonObjects");
+const ReadString = require('./ReadString');
 
-
 unit.add(module, [
-	function test_json_objects(t){
-		var async = t.startAsync("test_json_objects");
+  function test_json_objects(t) {
+    const async = t.startAsync('test_json_objects');
 
-		var stream  = StreamJsonObjects.make(),
-			pattern = [
-				1, 2, 3,
-				true, false,
-				"", "Abc",
-				[], [1], [1, []],
-				{}, {a: 1}, {b: {}, c: [{}]}
-			],
-			result = [];
+    const stream = StreamValues.withParser(),
+      pattern = [1, 2, 3, true, false, '', 'Abc', [], [1], [1, []], {}, {a: 1}, {b: {}, c: [{}]}],
+      result = [];
 
-		stream.output.on("data", function(data){
-			result[data.index] = data.value;
-		});
-		stream.output.on("end", function(){
-			eval(t.TEST("t.unify(pattern, result)"));
-			async.done();
-		});
+    stream.output.on('data', data => (result[data.key] = data.value));
+    stream.output.on('end', () => {
+      eval(t.TEST('t.unify(pattern, result)'));
+      async.done();
+    });
 
-		new ReadString(pattern.map(function (value) {
-			return JSON.stringify(value);
-		}).join(" ")).pipe(stream.input);
-	},
-	function test_no_json_objects (t) {
-		var async = t.startAsync("test_no_json_objects");
+    new ReadString(pattern.map(value => JSON.stringify(value)).join(' ')).pipe(stream.input);
+  },
+  function test_json_objects_no_streaming(t) {
+    const async = t.startAsync('test_json_objects_no_streaming');
 
-		var stream  = StreamJsonObjects.make(),
-			result  = [];
+    const stream = StreamValues.withParser({streamValues: false}),
+      pattern = [1, 2, 3, true, false, '', 'Abc', [], [1], [1, []], {}, {a: 1}, {b: {}, c: [{}]}],
+      result = [];
 
-		stream.output.on("data", function(data){
-			result[data.index] = data.value;
-		});
-		stream.output.on("end", function(){
-			eval(t.TEST("!result.length"));
-			async.done();
-		});
+    stream.output.on('data', data => (result[data.key] = data.value));
+    stream.output.on('end', () => {
+      eval(t.TEST('t.unify(pattern, result)'));
+      async.done();
+    });
 
-		new ReadString("").pipe(stream.input);
-	}
+    new ReadString(pattern.map(value => JSON.stringify(value)).join(' ')).pipe(stream.input);
+  },
+  function test_no_json_objects(t) {
+    const async = t.startAsync('test_no_json_objects');
+
+    const stream = StreamValues.withParser(),
+      result = [];
+
+    stream.on('data', data => (result[data.index] = data.value));
+    stream.on('end', () => {
+      eval(t.TEST('!result.length'));
+      async.done();
+    });
+
+    new ReadString('').pipe(stream);
+  },
+  function test_json_objects_filter(t) {
+    const async = t.startAsync('test_json_objects_filter');
+
+    const f = assembler => {
+      if (assembler.depth == 1 && assembler.key === null) {
+        if (assembler.current instanceof Array) {
+          return false; // reject
+        }
+        switch (assembler.current.a) {
+          case 'accept':
+            return true; // accept
+          case 'reject':
+            return false; // reject
+        }
+      }
+      // undecided
+    };
+
+    const stream = StreamValues.withParser({objectFilter: f}),
+      input = [
+        0,
+        1,
+        true,
+        false,
+        null,
+        {},
+        [],
+        {a: 'reject', b: [[[]]]},
+        ['c'],
+        {a: 'accept'},
+        {a: 'neutral'},
+        {x: true, a: 'reject'},
+        {y: null, a: 'accept'},
+        {z: 1234, a: 'neutral'},
+        {w: '12', a: 'neutral'}
+      ],
+      result = [];
+
+    stream.output.on('data', object => result.push(object.value));
+    stream.output.on('end', () => {
+      result.forEach(o => {
+        if (typeof o == 'object' && o) {
+          eval(t.TEST('!(o instanceof Array)'));
+          eval(t.TEST("o.a === 'accept'"));
+        } else {
+          eval(t.TEST('false')); // shouldn't be here
+        }
+      });
+      async.done();
+    });
+
+    new ReadString(input.map(value => JSON.stringify(value)).join(' ')).pipe(stream.input);
+  },
+  function test_json_objects_filter_include(t) {
+    const async = t.startAsync('test_json_objects_filter_include');
+
+    const f = assembler => {
+      if (assembler.depth == 1 && assembler.key === null) {
+        if (assembler.current instanceof Array) {
+          return false; // reject
+        }
+        switch (assembler.current.a) {
+          case 'accept':
+            return true; // accept
+          case 'reject':
+            return false; // reject
+        }
+      }
+      // undecided
+    };
+
+    const stream = StreamValues.withParser({objectFilter: f, includeUndecided: true}),
+      input = [
+        0,
+        1,
+        true,
+        false,
+        null,
+        {},
+        [],
+        {a: 'reject', b: [[[]]]},
+        ['c'],
+        {a: 'accept'},
+        {a: 'neutral'},
+        {x: true, a: 'reject'},
+        {y: null, a: 'accept'},
+        {z: 1234, a: 'neutral'},
+        {w: '12', a: 'neutral'}
+      ],
+      result = [];
+
+    stream.output.on('data', object => result.push(object.value));
+    stream.output.on('end', () => {
+      result.forEach(o => {
+        if (typeof o == 'object' && o) {
+          eval(t.TEST('!(o instanceof Array)'));
+          eval(t.TEST("o.a !== 'reject'"));
+        } else {
+          eval(t.TEST("o === null || typeof o != 'object'"));
+        }
+      });
+      async.done();
+    });
+
+    new ReadString(input.map(value => JSON.stringify(value)).join(' ')).pipe(stream.input);
+  }
 ]);

tests/test_object.js

@@ -1,59 +0,186 @@
-"use strict";
+'use strict';
 
+const unit = require('heya-unit');
 
-var unit = require("heya-unit");
+const ReadString = require('./ReadString');
+const StreamObject = require('../src/streamers/StreamObject');
 
-var ReadString  = require("./ReadString");
-var StreamObject = require("../utils/StreamObject");
+unit.add(module, [
+  function test_object(t) {
+    const async = t.startAsync('test_object');
 
+    const stream = StreamObject.withParser(),
+      pattern = {
+        str: 'bar',
+        baz: null,
+        t: true,
+        f: false,
+        zero: 0,
+        one: 1,
+        obj: {},
+        arr: [],
+        deepObj: {a: 'b'},
+        deepArr: ['c'],
+        '': '' // tricky, yet legal
+      },
+      result = {};
 
-unit.add(module, [
-	function test_object_fail(t){
-		var async = t.startAsync("test_object_fail");
+    stream.output.on('data', data => (result[data.key] = data.value));
+    stream.output.on('end', () => {
+      eval(t.TEST('t.unify(pattern, result)'));
+      async.done();
+    });
 
-		var stream = StreamObject.make();
+    new ReadString(JSON.stringify(pattern)).pipe(stream.input);
+  },
+  function test_object_fail(t) {
+    const async = t.startAsync('test_object_fail');
 
-		stream.output.on("data", function(value){
-			eval(t.TEST("!'We shouldn\'t be here.'"));
-		});
-		stream.output.on("error", function(err){
-			eval(t.TEST("err"));
-			async.done();
-		});
-		stream.output.on("end", function(value){
-			eval(t.TEST("!'We shouldn\'t be here.'"));
-			async.done();
-		});
+    const stream = StreamObject.withParser();
 
-		new ReadString(" true ").pipe(stream.input);
-	},
-	function test_object(t){
-		var async = t.startAsync("test_object");
+    stream.on('data', value => eval(t.TEST("!'We shouldn't be here.'")));
+    stream.on('error', err => {
+      eval(t.TEST('err'));
+      async.done();
+    });
+    stream.on('end', value => {
+      eval(t.TEST("!'We shouldn't be here.'"));
+      async.done();
+    });
 
-		var stream  = StreamObject.make(),
-			pattern = {
-				str: "bar",
-				baz: null,
-				t: true,
-				f: false,
-				zero: 0,
-				one: 1,
-				obj: {},
-				arr: [],
-				deepObj: {a: "b"},
-				deepArr: ["c"],
-				"": "" // tricky, yet legal
-			},
-			result = {};
+    new ReadString(' true ').pipe(stream);
+  },
+  function test_object_no_streaming(t) {
+    const async = t.startAsync('test_object_no_streaming');
 
-		stream.output.on("data", function(data){
-			result[data.key] = data.value;
-		});
-		stream.output.on("end", function(){
-			eval(t.TEST("t.unify(pattern, result)"));
-			async.done();
-		});
+    const stream = StreamObject.withParser({streamValues: false}),
+      pattern = {
+        str: 'bar',
+        baz: null,
+        t: true,
+        f: false,
+        zero: 0,
+        one: 1,
+        obj: {},
+        arr: [],
+        deepObj: {a: 'b'},
+        deepArr: ['c'],
+        '': '' // tricky, yet legal
+      },
+      result = {};
 
-		new ReadString(JSON.stringify(pattern)).pipe(stream.input);
-	}
+    stream.output.on('data', data => (result[data.key] = data.value));
+    stream.output.on('end', () => {
+      eval(t.TEST('t.unify(pattern, result)'));
+      async.done();
+    });
+
+    new ReadString(JSON.stringify(pattern)).pipe(stream.input);
+  },
+  function test_object_filter(t) {
+    const async = t.startAsync('test_object_filter');
+
+    const f = assembler => {
+      if (assembler.depth == 2 && assembler.key === null) {
+        if (assembler.current instanceof Array) {
+          return false; // reject
+        }
+        switch (assembler.current.a) {
+          case 'accept':
+            return true; // accept
+          case 'reject':
+            return false; // reject
+        }
+      }
+      // undecided
+    };
+
+    const stream = StreamObject.withParser({objectFilter: f}),
+      input = {
+        a: 0,
+        b: 1,
+        c: true,
+        d: false,
+        e: null,
+        f: {},
+        g: [],
+        h: {a: 'reject', b: [[[]]]},
+        i: ['c'],
+        j: {a: 'accept'},
+        k: {a: 'neutral'},
+        l: {x: true, a: 'reject'},
+        m: {y: null, a: 'accept'},
+        n: {z: 1234, a: 'neutral'},
+        o: {w: '12', a: 'neutral'}
+      },
+      result = [];
+
+    stream.output.on('data', object => result.push(object.value));
+    stream.output.on('end', () => {
+      result.forEach(o => {
+        if (typeof o == 'object' && o) {
+          eval(t.TEST('!(o instanceof Array)'));
+          eval(t.TEST("o.a === 'accept'"));
+        } else {
+          eval(t.TEST('false')); // shouldn't be here
+        }
+      });
+      async.done();
+    });
+
+    new ReadString(JSON.stringify(input)).pipe(stream.input);
+  },
+  function test_object_filter_include(t) {
+    const async = t.startAsync('test_object_filter_include');
+
+    const f = assembler => {
+      if (assembler.depth == 2 && assembler.key === null) {
+        if (assembler.current instanceof Array) {
+          return false; // reject
+        }
+        switch (assembler.current.a) {
+          case 'accept':
+            return true; // accept
+          case 'reject':
+            return false; // reject
+        }
+      }
+      // undecided
+    };
+
+    const stream = StreamObject.withParser({objectFilter: f, includeUndecided: true}),
+      input = {
+        a: 0,
+        b: 1,
+        c: true,
+        d: false,
+        e: null,
+        f: {},
+        g: [],
+        h: {a: 'reject', b: [[[]]]},
+        i: ['c'],
+        j: {a: 'accept'},
+        k: {a: 'neutral'},
+        l: {x: true, a: 'reject'},
+        m: {y: null, a: 'accept'},
+        n: {z: 1234, a: 'neutral'},
+        o: {w: '12', a: 'neutral'}
+      },
+      result = [];
+
+    stream.output.on('data', object => result.push(object.value));
+    stream.output.on('end', () => {
+      result.forEach(o => {
+        if (typeof o == 'object' && o) {
+          eval(t.TEST('!(o instanceof Array)'));
+          eval(t.TEST("o.a !== 'reject'"));
+        } else {
+          eval(t.TEST("o === null || typeof o != 'object'"));
+        }
+      });
+      async.done();
+    });
+
+    new ReadString(JSON.stringify(input)).pipe(stream.input);
+  }
 ]);

tests/test_parser.js

@@ -1,65 +0,283 @@
-"use strict";
+'use strict';
 
+const unit = require('heya-unit');
 
-var unit = require("heya-unit");
+const fs = require('fs'),
+  path = require('path'),
+  zlib = require('zlib');
 
-var ReadString = require("./ReadString");
-var Parser = require("../Parser");
+const Parser = require('../src/Parser');
+const Assembler = require('../src/Assembler');
+const emit = require('../src/utils/emit');
 
+const ReadString = require('./ReadString');
+const Counter = require('./Counter');
 
+const survivesRoundtrip = (t, object) => {
+  const async = t.startAsync('survivesRoundtrip: ' + object);
+
+  const input = JSON.stringify(object),
+    pipeline = new ReadString(input).pipe(new Parser()),
+    assembler = Assembler.connectTo(pipeline);
+
+  pipeline.on('end', () => {
+    eval(t.TEST('t.unify(assembler.current, object)'));
+    async.done();
+  });
+};
+
+function runSlidingWindowTest(t, quant) {
+  const async = t.startAsync('test_sliding: ' + quant);
+
+  const object = {
+      stringWithTabsAndNewlines: "Did it work?\nNo...\t\tI don't think so...",
+      anArray: [1, 2, true, 'tabs?\t\t\t\u0001\u0002\u0003', false]
+    },
+    input = JSON.stringify(object),
+    pipeline = new ReadString(input, quant).pipe(new Parser()),
+    assembler = Assembler.connectTo(pipeline);
+
+  pipeline.on('end', () => {
+    eval(t.TEST('t.unify(assembler.current, object)'));
+    async.done();
+  });
+}
+
 unit.add(module, [
-	function test_parser (t) {
-		var async = t.startAsync("test_parser");
+  function test_parser_streamer(t) {
+    const async = t.startAsync('test_streamer');
 
-		var input = '{"a": 1, "b": true, "c": ["d"]}',
-			pipeline = new ReadString(input).pipe(new Parser()),
-			result = [];
+    const input = '{"a": 1, "b": true, "c": ["d"]}',
+      pipeline = ReadString.make(input).pipe(Parser.make({packValues: false})),
+      result = [];
 
-		pipeline.on("data", function(chunk){
-			result.push({id: chunk.id, val: chunk.value});
-		});
-		pipeline.on("end", function(){
-			eval(t.ASSERT("result.length === 23"));
-			eval(t.TEST("result[0].id === '{' && result[0].val === '{'"));
-			eval(t.TEST("result[1].id === '\"' && result[1].val === '\"'"));
-			eval(t.TEST("result[2].id === 'plainChunk' && result[2].val === 'a'"));
-			eval(t.TEST("result[3].id === '\"' && result[3].val === '\"'"));
-			eval(t.TEST("result[4].id === ':' && result[4].val === ':'"));
-			eval(t.TEST("result[5].id === 'nonZero' && result[5].val === '1'"));
-			eval(t.TEST("result[6].id === ',' && result[6].val === ','"));
-			eval(t.TEST("result[7].id === '\"' && result[7].val === '\"'"));
-			eval(t.TEST("result[8].id === 'plainChunk' && result[8].val === 'b'"));
-			eval(t.TEST("result[9].id === '\"' && result[9].val === '\"'"));
-			eval(t.TEST("result[10].id === ':' && result[10].val === ':'"));
-			eval(t.TEST("result[11].id === 'true' && result[11].val === 'true'"));
-			eval(t.TEST("result[12].id === ',' && result[12].val === ','"));
-			eval(t.TEST("result[13].id === '\"' && result[13].val === '\"'"));
-			eval(t.TEST("result[14].id === 'plainChunk' && result[14].val === 'c'"));
-			eval(t.TEST("result[15].id === '\"' && result[15].val === '\"'"));
-			eval(t.TEST("result[16].id === ':' && result[16].val === ':'"));
-			eval(t.TEST("result[17].id === '[' && result[17].val === '['"));
-			eval(t.TEST("result[18].id === '\"' && result[18].val === '\"'"));
-			eval(t.TEST("result[19].id === 'plainChunk' && result[19].val === 'd'"));
-			eval(t.TEST("result[20].id === '\"' && result[20].val === '\"'"));
-			eval(t.TEST("result[21].id === ']' && result[21].val === ']'"));
-			eval(t.TEST("result[22].id === '}' && result[22].val === '}'"));
-			async.done();
-		});
-	},
-	function test_parser_fail (t) {
-		var async = t.startAsync("test_parser_fail");
+    pipeline.on('data', function(chunk) {
+      result.push({name: chunk.name, val: chunk.value});
+    });
+    pipeline.on('end', function() {
+      eval(t.ASSERT('result.length === 20'));
+      eval(t.TEST("result[0].name === 'startObject'"));
+      eval(t.TEST("result[1].name === 'startKey'"));
+      eval(t.TEST("result[2].name === 'stringChunk' && result[2].val === 'a'"));
+      eval(t.TEST("result[3].name === 'endKey'"));
+      eval(t.TEST("result[4].name === 'startNumber'"));
+      eval(t.TEST("result[5].name === 'numberChunk' && result[5].val === '1'"));
+      eval(t.TEST("result[6].name === 'endNumber'"));
+      eval(t.TEST("result[7].name === 'startKey'"));
+      eval(t.TEST("result[8].name === 'stringChunk' && result[8].val === 'b'"));
+      eval(t.TEST("result[9].name === 'endKey'"));
+      eval(t.TEST("result[10].name === 'trueValue' && result[10].val === true"));
+      eval(t.TEST("result[11].name === 'startKey'"));
+      eval(t.TEST("result[12].name === 'stringChunk' && result[12].val === 'c'"));
+      eval(t.TEST("result[13].name === 'endKey'"));
+      eval(t.TEST("result[14].name === 'startArray'"));
+      eval(t.TEST("result[15].name === 'startString'"));
+      eval(t.TEST("result[16].name === 'stringChunk' && result[16].val === 'd'"));
+      eval(t.TEST("result[17].name === 'endString'"));
+      eval(t.TEST("result[18].name === 'endArray'"));
+      eval(t.TEST("result[19].name === 'endObject'"));
+      async.done();
+    });
+  },
+  function test_parser_packer(t) {
+    const async = t.startAsync('test_parser_packer');
 
-		var stream = new Parser();
+    const input = '{"a": 1, "b": true, "c": ["d"]}',
+      pipeline = new ReadString(input).pipe(new Parser()),
+      result = [];
 
-		stream.on("error", function (err) {
-			eval(t.TEST("err"));
-			async.done();
-		});
-		stream.on("end", function (value) {
-			eval(t.TEST("!'We shouldn\'t be here.'"));
-			async.done();
-		});
+    pipeline.on('data', chunk => result.push({name: chunk.name, val: chunk.value}));
+    pipeline.on('end', () => {
+      eval(t.ASSERT('result.length === 25'));
+      eval(t.TEST("result[0].name === 'startObject'"));
+      eval(t.TEST("result[1].name === 'startKey'"));
+      eval(t.TEST("result[2].name === 'stringChunk' && result[2].val === 'a'"));
+      eval(t.TEST("result[3].name === 'endKey'"));
+      eval(t.TEST("result[4].name === 'keyValue' && result[4].val === 'a'"));
+      eval(t.TEST("result[5].name === 'startNumber'"));
+      eval(t.TEST("result[6].name === 'numberChunk' && result[6].val === '1'"));
+      eval(t.TEST("result[7].name === 'endNumber'"));
+      eval(t.TEST("result[8].name === 'numberValue' && result[8].val === '1'"));
+      eval(t.TEST("result[9].name === 'startKey'"));
+      eval(t.TEST("result[10].name === 'stringChunk' && result[10].val === 'b'"));
+      eval(t.TEST("result[11].name === 'endKey'"));
+      eval(t.TEST("result[12].name === 'keyValue' && result[12].val === 'b'"));
+      eval(t.TEST("result[13].name === 'trueValue' && result[13].val === true"));
+      eval(t.TEST("result[14].name === 'startKey'"));
+      eval(t.TEST("result[15].name === 'stringChunk' && result[15].val === 'c'"));
+      eval(t.TEST("result[16].name === 'endKey'"));
+      eval(t.TEST("result[17].name === 'keyValue' && result[17].val === 'c'"));
+      eval(t.TEST("result[18].name === 'startArray'"));
+      eval(t.TEST("result[19].name === 'startString'"));
+      eval(t.TEST("result[20].name === 'stringChunk' && result[20].val === 'd'"));
+      eval(t.TEST("result[21].name === 'endString'"));
+      eval(t.TEST("result[22].name === 'stringValue' && result[22].val === 'd'"));
+      eval(t.TEST("result[23].name === 'endArray'"));
+      eval(t.TEST("result[24].name === 'endObject'"));
+      async.done();
+    });
+  },
+  function test_parser_packer_no_streamed_values(t) {
+    const async = t.startAsync('test_parser_packer_no_streamed_values');
 
-		new ReadString("{").pipe(stream);
-	}
+    const input = '{"a": 1, "b": true, "c": ["d"]}',
+      pipeline = new ReadString(input).pipe(new Parser({streamValues: false})),
+      result = [];
+
+    pipeline.on('data', chunk => result.push({name: chunk.name, val: chunk.value}));
+    pipeline.on('end', () => {
+      eval(t.ASSERT('result.length === 10'));
+      eval(t.TEST("result[0].name === 'startObject'"));
+      eval(t.TEST("result[1].name === 'keyValue' && result[1].val === 'a'"));
+      eval(t.TEST("result[2].name === 'numberValue' && result[2].val === '1'"));
+      eval(t.TEST("result[3].name === 'keyValue' && result[3].val === 'b'"));
+      eval(t.TEST("result[4].name === 'trueValue' && result[4].val === true"));
+      eval(t.TEST("result[5].name === 'keyValue' && result[5].val === 'c'"));
+      eval(t.TEST("result[6].name === 'startArray'"));
+      eval(t.TEST("result[7].name === 'stringValue' && result[7].val === 'd'"));
+      eval(t.TEST("result[8].name === 'endArray'"));
+      eval(t.TEST("result[9].name === 'endObject'"));
+      async.done();
+    });
+  },
+  function test_parser_emitter(t) {
+    const async = t.startAsync('test_parser_emitter');
+
+    const plainCounter = new Counter(),
+      emitterCounter = new Counter(),
+      parser = emit(new Parser({streamValues: false}));
+
+    parser.on('startObject', () => ++emitterCounter.objects);
+    parser.on('keyValue', () => ++emitterCounter.keys);
+    parser.on('startArray', () => ++emitterCounter.arrays);
+    parser.on('nullValue', () => ++emitterCounter.nulls);
+    parser.on('trueValue', () => ++emitterCounter.trues);
+    parser.on('falseValue', () => ++emitterCounter.falses);
+    parser.on('numberValue', () => ++emitterCounter.numbers);
+    parser.on('stringValue', () => ++emitterCounter.strings);
+
+    parser.on('finish', () => {
+      eval(t.TEST('t.unify(plainCounter, emitterCounter)'));
+      async.done();
+    });
+
+    fs.readFile(path.resolve(__dirname, './sample.json.gz'), (err, data) => {
+      if (err) {
+        throw err;
+      }
+      zlib.gunzip(data, (err, data) => {
+        if (err) {
+          throw err;
+        }
+
+        const o = JSON.parse(data);
+        Counter.walk(o, plainCounter);
+
+        fs.createReadStream(path.resolve(__dirname, './sample.json.gz'))
+          .pipe(zlib.createGunzip())
+          .pipe(parser);
+      });
+    });
+  },
+  function test_parser_escaped(t) {
+    const async = t.startAsync('test_escaped');
+
+    const object = {
+        stringWithTabsAndNewlines: "Did it work?\nNo...\t\tI don't think so...",
+        anArray: [1, 2, true, 'tabs?\t\t\t\u0001\u0002\u0003', false]
+      },
+      input = JSON.stringify(object),
+      pipeline = new ReadString(input).pipe(new Parser()),
+      assembler = Assembler.connectTo(pipeline);
+
+    pipeline.on('end', () => {
+      eval(t.TEST('t.unify(assembler.current, object)'));
+      async.done();
+    });
+  },
+  function test_parser_primitives_true(t) {
+    survivesRoundtrip(t, true);
+  },
+  function test_parser_primitives_false(t) {
+    survivesRoundtrip(t, false);
+  },
+  function test_parser_primitives_null(t) {
+    survivesRoundtrip(t, null);
+  },
+  function test_parser_primitives_number1(t) {
+    survivesRoundtrip(t, 0);
+  },
+  function test_parser_primitives_number2(t) {
+    survivesRoundtrip(t, -1);
+  },
+  function test_parser_primitives_number3(t) {
+    survivesRoundtrip(t, 1.5);
+  },
+  function test_parser_primitives_number4(t) {
+    survivesRoundtrip(t, 1.5e-12);
+  },
+  function test_parser_primitives_number5(t) {
+    survivesRoundtrip(t, 1.5e33);
+  },
+  function test_parser_primitives_string(t) {
+    survivesRoundtrip(t, 'string');
+  },
+  function test_parser_primitives_empty_object(t) {
+    survivesRoundtrip(t, {});
+  },
+  function test_parser_primitives_empty_array(t) {
+    survivesRoundtrip(t, []);
+  },
+  function test_parser_sliding_1(t) {
+    runSlidingWindowTest(t, 1);
+  },
+  function test_parser_sliding_2(t) {
+    runSlidingWindowTest(t, 2);
+  },
+  function test_parser_sliding_3(t) {
+    runSlidingWindowTest(t, 3);
+  },
+  function test_parser_sliding_4(t) {
+    runSlidingWindowTest(t, 4);
+  },
+  function test_parser_sliding_5(t) {
+    runSlidingWindowTest(t, 5);
+  },
+  function test_parser_sliding_6(t) {
+    runSlidingWindowTest(t, 6);
+  },
+  function test_parser_sliding_7(t) {
+    runSlidingWindowTest(t, 7);
+  },
+  function test_parser_sliding_8(t) {
+    runSlidingWindowTest(t, 8);
+  },
+  function test_parser_sliding_9(t) {
+    runSlidingWindowTest(t, 9);
+  },
+  function test_parser_sliding_10(t) {
+    runSlidingWindowTest(t, 10);
+  },
+  function test_parser_sliding_11(t) {
+    runSlidingWindowTest(t, 11);
+  },
+  function test_parser_sliding_12(t) {
+    runSlidingWindowTest(t, 12);
+  },
+  function test_parser_fail(t) {
+    const async = t.startAsync('test_parser_fail');
+
+    const stream = new Parser();
+
+    stream.on('error', err => {
+      eval(t.TEST('err'));
+      async.done();
+    });
+    stream.on('end', value => {
+      eval(t.TEST("!'We shouldn't be here.'"));
+      async.done();
+    });
+
+    new ReadString('{').pipe(stream);
+  }
 ]);

tests/test_primitives.js

@@ -1,66 +0,56 @@
-"use strict";
+'use strict';
 
+const unit = require('heya-unit');
 
-var unit = require("heya-unit");
+const Assembler = require('../src/Assembler');
+const Parser = require('../src/Parser');
+const ReadString = require('./ReadString');
 
-var Assembler = require("../utils/Assembler");
+const survivesRoundtrip = (t, object) => {
+  const async = t.startAsync('survivesRoundtrip: ' + object);
 
-var Parser   = require("../Parser");
-var Streamer = require("../Streamer");
-var Packer   = require("../Packer");
+  const input = JSON.stringify(object),
+    pipeline = new ReadString(input).pipe(new Parser()),
+    assembler = Assembler.connectTo(pipeline);
 
-var ReadString = require("./ReadString");
+  pipeline.on('end', () => {
+    eval(t.TEST('t.unify(assembler.current, object)'));
+    async.done();
+  });
+};
 
-
-function survivesRoundtrip(t, object){
-	var async = t.startAsync("survivesRoundtrip: " + object);
-
-	var input = JSON.stringify(object),
-		pipeline = new ReadString(input).pipe(new Parser()).pipe(new Streamer()).
-			pipe(new Packer({packKeys: true, packStrings: true, packNumbers: true})),
-		assembler = new Assembler();
-
-	pipeline.on("data", function(chunk){
-		assembler[chunk.name] && assembler[chunk.name](chunk.value);
-	});
-	pipeline.on("end", function(){
-		eval(t.TEST("t.unify(assembler.current, object)"));
-		async.done();
-	});
-}
-
 unit.add(module, [
-	function test_primitives_true(t){
-		survivesRoundtrip(t, true);
-	},
-	function test_primitives_false(t){
-		survivesRoundtrip(t, false);
-	},
-	function test_primitives_null(t){
-		survivesRoundtrip(t, null);
-	},
-	function test_primitives_number1(t){
-		survivesRoundtrip(t, 0);
-	},
-	function test_primitives_number2(t){
-		survivesRoundtrip(t, -1);
-	},
-	function test_primitives_number3(t){
-		survivesRoundtrip(t, 1.5);
-	},
-	function test_primitives_number4(t){
-		survivesRoundtrip(t, 1.5e-12);
-	},
-	function test_primitives_number5(t){
-		survivesRoundtrip(t, 1.5e+33);
-	},
-	function test_primitives_string(t){
-		survivesRoundtrip(t, "string");
-	},
-	function test_primitives_empty_object(t){
-		survivesRoundtrip(t, {});
-	},
-	function test_primitives_empty_array(t){
-		survivesRoundtrip(t, []);
-	}
+  function test_primitives_true(t) {
+    survivesRoundtrip(t, true);
+  },
+  function test_primitives_false(t) {
+    survivesRoundtrip(t, false);
+  },
+  function test_primitives_null(t) {
+    survivesRoundtrip(t, null);
+  },
+  function test_primitives_number1(t) {
+    survivesRoundtrip(t, 0);
+  },
+  function test_primitives_number2(t) {
+    survivesRoundtrip(t, -1);
+  },
+  function test_primitives_number3(t) {
+    survivesRoundtrip(t, 1.5);
+  },
+  function test_primitives_number4(t) {
+    survivesRoundtrip(t, 1.5e-12);
+  },
+  function test_primitives_number5(t) {
+    survivesRoundtrip(t, 1.5e33);
+  },
+  function test_primitives_string(t) {
+    survivesRoundtrip(t, 'string');
+  },
+  function test_primitives_empty_object(t) {
+    survivesRoundtrip(t, {});
+  },
+  function test_primitives_empty_array(t) {
+    survivesRoundtrip(t, []);
+  }
 ]);

tests/test_sliding.js

@@ -1,73 +0,63 @@
-"use strict";
+'use strict';
 
+const unit = require('heya-unit');
 
-var unit = require("heya-unit");
+const Assembler = require('../src/Assembler');
+const Parser = require('../src/Parser');
+const ReadString = require('./ReadString');
 
-var Assembler = require("../utils/Assembler");
+const runSlidingWindowTest = (t, quant) => {
+  const async = t.startAsync('test_sliding: ' + quant);
 
-var Parser   = require("../Parser");
-var Streamer = require("../Streamer");
-var Packer   = require("../Packer");
+  const object = {
+      stringWithTabsAndNewlines: "Did it work?\nNo...\t\tI don't think so...",
+      anArray: [1, 2, true, 'tabs?\t\t\t\u0001\u0002\u0003', false]
+    },
+    input = JSON.stringify(object),
+    pipeline = new ReadString(input, quant).pipe(new Parser()),
+    assembler = Assembler.connectTo(pipeline);
 
-var ReadString = require("./ReadString");
+  pipeline.on('end', () => {
+    eval(t.TEST('t.unify(assembler.current, object)'));
+    async.done();
+  });
+};
 
-
-function runSlidingWindowTest(t, quant){
-	var async = t.startAsync("test_sliding: " + quant);
-
-	var object = {
-			stringWithTabsAndNewlines: "Did it work?\nNo...\t\tI don't think so...",
-			anArray: [1, 2, true, "tabs?\t\t\t\u0001\u0002\u0003", false]
-		},
-		input = JSON.stringify(object),
-		pipeline = new ReadString(input, quant).pipe(new Parser()).pipe(new Streamer()).
-			pipe(new Packer({packKeys: true, packStrings: true, packNumbers: true})),
-		assembler = new Assembler();
-
-	pipeline.on("data", function(chunk){
-		assembler[chunk.name] && assembler[chunk.name](chunk.value);
-	});
-	pipeline.on("end", function(){
-		eval(t.TEST("t.unify(assembler.current, object)"));
-		async.done();
-	});
-}
-
 unit.add(module, [
-	function test_sliding_1(t){
-		runSlidingWindowTest(t, 1);
-	},
-	function test_sliding_2(t){
-		runSlidingWindowTest(t, 2);
-	},
-	function test_sliding_3(t){
-		runSlidingWindowTest(t, 3);
-	},
-	function test_sliding_4(t){
-		runSlidingWindowTest(t, 4);
-	},
-	function test_sliding_5(t){
-		runSlidingWindowTest(t, 5);
-	},
-	function test_sliding_6(t){
-		runSlidingWindowTest(t, 6);
-	},
-	function test_sliding_7(t){
-		runSlidingWindowTest(t, 7);
-	},
-	function test_sliding_8(t){
-		runSlidingWindowTest(t, 8);
-	},
-	function test_sliding_9(t){
-		runSlidingWindowTest(t, 9);
-	},
-	function test_sliding_10(t){
-		runSlidingWindowTest(t, 10);
-	},
-	function test_sliding_11(t){
-		runSlidingWindowTest(t, 11);
-	},
-	function test_sliding_12(t){
-		runSlidingWindowTest(t, 12);
-	}
+  function test_sliding_1(t) {
+    runSlidingWindowTest(t, 1);
+  },
+  function test_sliding_2(t) {
+    runSlidingWindowTest(t, 2);
+  },
+  function test_sliding_3(t) {
+    runSlidingWindowTest(t, 3);
+  },
+  function test_sliding_4(t) {
+    runSlidingWindowTest(t, 4);
+  },
+  function test_sliding_5(t) {
+    runSlidingWindowTest(t, 5);
+  },
+  function test_sliding_6(t) {
+    runSlidingWindowTest(t, 6);
+  },
+  function test_sliding_7(t) {
+    runSlidingWindowTest(t, 7);
+  },
+  function test_sliding_8(t) {
+    runSlidingWindowTest(t, 8);
+  },
+  function test_sliding_9(t) {
+    runSlidingWindowTest(t, 9);
+  },
+  function test_sliding_10(t) {
+    runSlidingWindowTest(t, 10);
+  },
+  function test_sliding_11(t) {
+    runSlidingWindowTest(t, 11);
+  },
+  function test_sliding_12(t) {
+    runSlidingWindowTest(t, 12);
+  }
 ]);

tests/test_stringer.js

@@ -1,79 +0,154 @@
-"use strict";
+'use strict';
 
+const unit = require('heya-unit');
 
-var unit = require("heya-unit");
+const ReadString = require('./ReadString');
+const makeParser = require('../src/main');
+const Stringer = require('../src/Stringer');
 
-var ReadString = require("./ReadString");
-var makeSource = require("../main");
-var Stringer   = require("../utils/Stringer");
+unit.add(module, [
+  function test_stringer(t) {
+    const async = t.startAsync('test_stringer');
 
+    const parser = makeParser(),
+      stringer = new Stringer(),
+      pattern = {
+        a: [[[]]],
+        b: {a: 1},
+        c: {a: 1, b: 2},
+        d: [true, 1, "'x\"y'", null, false, true, {}, [], ''],
+        e: 1,
+        f: '',
+        g: true,
+        h: false,
+        i: null,
+        j: [],
+        k: {}
+      },
+      string = JSON.stringify(pattern);
+    let buffer = '';
 
-unit.add(module, [
-	function test_stringer (t) {
-		var async = t.startAsync("test_stringer");
+    parser.pipe(stringer);
 
-		var source   = makeSource(),
-			stringer = new Stringer(),
-			pattern  = {
-				a: [[[]]],
-				b: {a: 1},
-				c: {a: 1, b: 2},
-				d: [true, 1, "'x\"y'", null, false, true, {}, [], ""],
-				e: 1,
-				f: "",
-				g: true,
-				h: false,
-				i: null,
-				j: [],
-				k: {}
-			},
-			string = JSON.stringify(pattern),
-			buffer = '';
+    stringer.on('data', data => (buffer += data));
+    stringer.on('end', () => {
+      eval(t.TEST('string === buffer'));
+      async.done();
+    });
 
-		source.output.pipe(stringer);
+    new ReadString(string).pipe(parser);
+  },
+  function test_stringer_no_packing(t) {
+    const async = t.startAsync('test_stringer');
 
-		stringer.on("data", function(data) {
-			buffer += data;
-		});
-		stringer.on("end", function() {
-			eval(t.TEST('string === buffer'));
-			async.done();
-		});
+    const parser = makeParser({packValues: false}),
+      stringer = new Stringer(),
+      pattern = {
+        a: [[[]]],
+        b: {a: 1},
+        c: {a: 1, b: 2},
+        d: [true, 1, "'x\"y'", null, false, true, {}, [], ''],
+        e: 1,
+        f: '',
+        g: true,
+        h: false,
+        i: null,
+        j: [],
+        k: {}
+      },
+      string = JSON.stringify(pattern);
+    let buffer = '';
 
-		new ReadString(string).pipe(source.input);
-	},
-	function test_stringer_json_stream (t) {
-		var async = t.startAsync("test_stringer_json_stream");
+    parser.pipe(stringer);
 
-		var source   = makeSource({jsonStreaming: true}),
-			stringer = new Stringer(),
-			pattern  = {
-				a: [[[]]],
-				b: {a: 1},
-				c: {a: 1, b: 2},
-				d: [true, 1, "'x\"y'", null, false, true, {}, [], ""],
-				e: 1,
-				f: "",
-				g: true,
-				h: false,
-				i: null,
-				j: [],
-				k: {}
-			},
-			string = JSON.stringify(pattern),
-			buffer = '';
-		string += string;
+    stringer.on('data', data => (buffer += data));
+    stringer.on('end', () => {
+      eval(t.TEST('string === buffer'));
+      async.done();
+    });
 
-		source.output.pipe(stringer);
+    new ReadString(string).pipe(parser);
+  },
+  function test_stringer_values(t) {
+    const async = t.startAsync('test_stringer_values');
 
-		stringer.on("data", function(data) {
-			buffer += data;
-		});
-		stringer.on("end", function() {
-			eval(t.TEST('string === buffer'));
-			async.done();
-		});
+    const parser = makeParser(),
+      stringer = new Stringer({useValues: true}),
+      pattern = {
+        a: [[[]]],
+        b: {a: 1},
+        c: {a: 1, b: 2},
+        d: [true, 1, "'x\"y'", null, false, true, {}, [], ''],
+        e: 1,
+        f: '',
+        g: true,
+        h: false,
+        i: null,
+        j: [],
+        k: {}
+      },
+      string = JSON.stringify(pattern);
+    let buffer = '';
 
-		new ReadString(string).pipe(source.input);
-	}
+    parser.pipe(stringer);
+
+    stringer.on('data', data => (buffer += data));
+    stringer.on('end', () => {
+      eval(t.TEST('string === buffer'));
+      async.done();
+    });
+
+    new ReadString(string).pipe(parser);
+  },
+  function test_stringer_json_stream_objects(t) {
+    const async = t.startAsync('test_stringer_json_stream_objects');
+
+    const parser = makeParser({jsonStreaming: true}),
+      stringer = new Stringer(),
+      pattern = {
+        a: [[[]]],
+        b: {a: 1},
+        c: {a: 1, b: 2},
+        d: [true, 1, "'x\"y'", null, false, true, {}, [], ''],
+        e: 1,
+        f: '',
+        g: true,
+        h: false,
+        i: null,
+        j: [],
+        k: {}
+      };
+    let string = JSON.stringify(pattern),
+      buffer = '';
+
+    string += string;
+
+    parser.pipe(stringer);
+
+    stringer.on('data', data => (buffer += data));
+    stringer.on('end', () => {
+      eval(t.TEST('string === buffer'));
+      async.done();
+    });
+
+    new ReadString(string).pipe(parser);
+  },
+  function test_stringer_json_stream_primitives(t) {
+    const async = t.startAsync('test_stringer_json_stream_primitives');
+
+    const parser = makeParser({jsonStreaming: true}),
+      stringer = new Stringer(),
+      string = '1 2 "zzz" "z\'z\\"z" null true false 1[]null{}true';
+    let buffer = '';
+
+    parser.pipe(stringer);
+
+    stringer.on('data', data => (buffer += data));
+    stringer.on('end', () => {
+      eval(t.TEST('string === buffer'));
+      async.done();
+    });
+
+    new ReadString(string).pipe(parser);
+  }
 ]);

tests/tests.js

@@ -1,27 +0,21 @@
-"use strict";
+'use strict';
 
+const unit = require('heya-unit');
 
-var unit = require("heya-unit");
+require('./test_parser');
+require('./test_main');
+require('./test_filter');
+require('./test_escaped');
+require('./test_emitter');
+require('./test_assembler');
+require('./test_stringer');
+require('./test_primitives');
+require('./test_sliding');
+require('./test_array');
+require('./test_object');
+require('./test_json_stream');
+require('./test_pick');
+require('./test_replace');
+require('./test_ignore');
 
-require("./test_classic");
-require("./test_alternative");
-require("./test_parser");
-require("./test_streamer");
-require("./test_packer");
-require("./test_filter");
-require("./test_escaped");
-require("./test_source");
-require("./test_emitter");
-require("./test_assembler");
-require("./test_stringer");
-require("./test_primitives");
-require("./test_sliding");
-require("./test_array");
-require("./test_filtered_array");
-require("./test_filter_objects");
-require("./test_combo");
-require("./test_object");
-require("./test_json_stream");
-
-
 unit.run();

New alerts

License Policy Violation

License

This package is not allowed per your license policy. Review the package's license to ensure compliance.

Found 1 instance in 1 package

Major refactor

Supply chain risk

Package has recently undergone a major refactor. It may be unstable or indicate significant internal changes. Use caution when updating to versions that include significant changes.

Found 1 instance in 1 package

Filesystem access

Supply chain risk

Accesses the file system, and could potentially read sensitive data.

Found 1 instance in 1 package

Fixed alerts

License Policy Violation

License

This package is not allowed per your license policy. Review the package's license to ensure compliance.

Found 1 instance in 1 package

Dynamic require

Supply chain risk

Dynamic require can indicate the package is performing dangerous or unsafe dynamic code execution.

Found 1 instance in 1 package

Filesystem access

Supply chain risk

Accesses the file system, and could potentially read sensitive data.

Found 1 instance in 1 package

No v1

Quality

Package is not semver >=1. This means it is not stable and does not support ^ ranges.

Found 1 instance in 1 package

Improved metrics

Number of low quality alerts

0

decreased by-100%

Number of low supply chain risk alerts

4

decreased by-60%

Number of medium supply chain risk alerts

214

decreased by-14.4%

Worsened metrics

Total package byte prevSize

175682

decreased by-21.19%

Number of package files

42

decreased by-28.81%

Lines of code

3338

decreased by-16.15%

Number of lines in readme file

124

decreased by-83.36%

Dependency changes

• + Addedstream-chain@^2.0.2

• + Addedstream-chain@2.2.5(transitive)

• - Removedparser-toolkit@>=0.0.3

• - Removedparser-toolkit@0.0.5(transitive)