jsonld-streaming-parser

JSON-LD Streaming Parser

A fast and lightweight streaming and 100% spec-compliant JSON-LD 1.1 parser, with RDFJS representations of RDF terms, quads and triples.

The streaming nature allows triples to be emitted as soon as possible, and documents larger than memory to be parsed.

Make sure to enable the streamingProfile flag when parsing a JSON-LD document with a streaming profile to exploit the streaming capabilities of this parser, as this is disabled by default.

Installation

$ npm install jsonld-streaming-parser

or

$ yarn add jsonld-streaming-parser

This package also works out-of-the-box in browsers via tools such as webpack and browserify.

Require

import {JsonLdParser} from "jsonld-streaming-parser";

or

const JsonLdParser = require("jsonld-streaming-parser").JsonLdParser;

Usage

JsonLdParser is a Node Transform stream that takes in chunks of JSON-LD data, and outputs RDFJS-compliant quads.

It can be used to pipe streams to, or you can write strings into the parser directly.

Print all parsed triples from a file to the console

const myParser = new JsonLdParser();

fs.createReadStream('myfile.jsonld')
  .pipe(myParser)
  .on('data', console.log)
  .on('error', console.error)
  .on('end', () => console.log('All triples were parsed!'));

Manually write strings to the parser

const myParser = new JsonLdParser();

myParser
  .on('data', console.log)
  .on('error', console.error)
  .on('end', () => console.log('All triples were parsed!'));

myParser.write('{');
myParser.write(`"@context": "https://schema.org/",`);
myParser.write(`"@type": "Recipe",`);
myParser.write(`"name": "Grandma's Holiday Apple Pie",`);
myParser.write(`"aggregateRating": {`);
myParser.write(`"@type": "AggregateRating",`);
myParser.write(`"ratingValue": "4"`);
myParser.write(`}}`);
myParser.end();

Import streams

This parser implements the RDFJS Sink interface, which makes it possible to alternatively parse streams using the import method.

const myParser = new JsonLdParser();

const myTextStream = fs.createReadStream('myfile.jsonld');

myParser.import(myTextStream)
  .on('data', console.log)
  .on('error', console.error)
  .on('end', () => console.log('All triples were parsed!'));

Capture detected contexts

Using a context event listener, you can collect all detected contexts.

const myParser = new JsonLdParser();

const myTextStream = fs.createReadStream('myfile.jsonld');

myParser.import(myTextStream)
  .on('context', console.log)
  .on('data', console.error)
  .on('error', console.error)
  .on('end', () => console.log('All triples were parsed!'));

Parse from HTTP responses

Usually, JSON-LD is published via the application/ld+json media type. However, when a JSON-LD context is attached via a link header, then it can also be published via application/json and +json extension types.

This library exposes the JsonLdParser.fromHttpResponse function to abstract these cases, so that you can call it for any HTTP response, and it will return an appropriate parser which may or may not contain a custom header-defined context:

const myParser = JsonLdParser.fromHttpResponse(
  'http://example.org/my-file.json', // For example: response.url
  'application/json', // For example: headers.get('content-type')
  new Headers({ 'Link': '<my-context.jsonld>; rel=\"http://www.w3.org/ns/json-ld#context\"' }), // Optional: WHATWG Headers 
  {}, // Optional: Any options you want to pass to the parser
);

// Parse anything with myParser like usual
const quads = myParser.import(response.body);

The Headers object must implement the Headers interface from the WHATWG Fetch API.

This function will automatically detect the http://www.w3.org/ns/json-ld#streaming profile and set the streamingProfile flag.

Configuration

Optionally, the following parameters can be set in the JsonLdParser constructor:

• dataFactory: A custom RDFJS DataFactory to construct terms and triples. (Default: require('@rdfjs/data-model'))
• context: An optional root context to use while parsing. This can by anything that is accepted by jsonld-context-parser, such as a URL, object or array. (Default: {})
• baseIRI: An initial default base IRI. (Default: '')
• streamingProfile: If this parser can assume that parsed documents follow the streaming JSON-LD profile. If true, and a non-streaming document is detected, an error may be thrown. If false, non-streaming documents will be handled by preemptively buffering entries, which will lose many of the streaming benefits of this parser. (Default: true)
• documentLoader A custom loader for fetching remote contexts. This can be set to anything that implements IDocumentLoader (Default: FetchDocumentLoader)
• produceGeneralizedRdf: If blank node predicates should be allowed, they will be ignored otherwise. (Default: false)
• processingMode: The maximum JSON-LD version that should be processable by this parser. (Default: 1.0)
• strictValues: By default, JSON-LD requires that all properties (or @id's) that are not URIs, are unknown keywords, and do not occur in the context should be silently dropped. When setting this value to true, an error will be thrown when such properties occur. This also applies to invalid values such as language tags. This is useful for debugging JSON-LD documents. (Default: false)
• allowSubjectList: If RDF lists can appear in the subject position. (Default: false)
• validateValueIndexes: If @index inside array nodes should be validated. I.e., nodes inside the same array with the same @id, should have equal @index values. This is not applicable to this parser as we don't do explicit flattening, but it is required to be spec-compliant. (Default: false)
• defaultGraph: The default graph for constructing quads. (Default: defaultGraph())
• rdfDirection: The mode under which @direction should be handled. If undefined, @direction is ignored. Alternatively, it can be set to either 'i18n-datatype' or 'compound-literal' (Default: undefined)
• normalizeLanguageTags: Whether or not language tags should be normalized to lowercase. (Default: false for JSON-LD 1.1 (and higher), true for JSON-LD 1.0)
• streamingProfileAllowOutOfOrderPlainType: When the streaming profile flag is enabled, @type entries MUST come before other properties since they may defined a type-scoped context. However, when this flag is enabled, @type entries that do NOT define a type-scoped context may appear anywhere just like a regular property.. (Default: false)

new JsonLdParser({
  dataFactory: require('@rdfjs/data-model'),
  context: 'https://schema.org/',
  baseIRI: 'http://example.org/',
  streamingProfile: true,
  documentLoader: new FetchDocumentLoader(),
  produceGeneralizedRdf: false,
  processingMode: '1.0',
  errorOnInvalidIris: false,
  allowSubjectList: false,
  validateValueIndexes: false,
  defaultGraph: namedNode('http://example.org/graph'),
  rdfDirection: 'i18n-datatype',
  normalizeLanguageTags: true,
});

How it works

This parser does not follow the recommended procedure for transforming JSON-LD to RDF, because this does not allow stream-based handling of JSON. Instead, this tool introduces an alternative streaming algorithm that achieves spec-compliant JSON-LD parsing.

This parser builds on top of the jsonparse library, which is a sax-based streaming JSON parser. With this, several in-memory stacks are maintained. These stacks are needed to accumulate the required information to emit triples/quads. These stacks are deleted from the moment they are not needed anymore, to limit memory usage.

The algorithm makes a couple of (soft) assumptions regarding the structure of the JSON-LD document, which is true for most typical JSON-LD documents.

• If there is a @context, it is the first entry of an object.
• If there is an @id, it comes right after @context, or is the first entry of an object.

If these assumptions are met, (almost) each object entry corresponds to a triple/quad that can be emitted. For example, the following document allows a triple to be emitted after each object entry (except for first two lines):

{
  "@context": "http://schema.org/",
  "@id": "http://example.org/",
  "@type": "Person",               // --> <http://example.org/> a schema:Person.
  "name": "Jane Doe",              // --> <http://example.org/> schema:name "Jane Doe".
  "jobTitle": "Professor",         // --> <http://example.org/> schema:jobTitle "Professor".
  "telephone": "[phone]",   // --> <http://example.org/> schema:telephone "[phone]".
  "url": "http://www.janedoe.com"  // --> <http://example.org/> schema:url <http://www.janedoe.com>.
}

If not all of these assumptions are met, entries of an object are buffered until enough information becomes available, or if the object is closed. For example, if no @id was present, values will be buffered until an @id is read, or if the object closed.

For example:

{
  "@context": "http://schema.org/",
  "@type": "Person",
  "name": "Jane Doe",
  "jobTitle": "Professor",
  "@id": "http://example.org/",    // --> <http://example.org/> a schema:Person.
                                   // --> <http://example.org/> schema:name "Jane Doe".
                                   // --> <http://example.org/> schema:jobTitle "Professor".
  "telephone": "[phone]",   // --> <http://example.org/> schema:telephone "[phone]".
  "url": "http://www.janedoe.com"  // --> <http://example.org/> schema:url <http://www.janedoe.com>.
}

As such, JSON-LD documents that meet these requirements will be parsed very efficiently. Other documents will still be parsed correctly as well, with a slightly lower efficiency.

Streaming Profile

This parser adheres to both the JSON-LD 1.1 specification and the JSON-LD 1.1 Streaming specification.

By default, this parser assumes that JSON-LD document are not in the streaming document form. This means that the parser may buffer large parts of the document before quads are produced, to make sure that the document is interpreted correctly.

Since this buffering neglects the streaming benefits of this parser, the streamingProfile flag should be enabled when a streaming JSON-LD document is being parsed.

If non-streaming JSON-LD documents are encountered when the streamingProfile flag is enabled, an error may be thrown.

Specification compliance

This parser implements the following JSON-LD specifications:

• JSON-LD 1.1 - Transform JSON-LD to RDF
• JSON-LD 1.1 - Error handling
• JSON-LD 1.1 - Streaming Transform JSON-LD to RDF

Performance

The following table shows some simple performance comparisons between JSON-LD Streaming Parser and jsonld.js.

These basic experiments show that even though streaming parsers are typically significantly slower than regular parsers, JSON-LD Streaming Parser still achieves similar performance as jsonld.js for most typical JSON-LD files. However, for expanded JSON-LD documents, JSON-LD Streaming Parser is around 3~4 times slower.

File	JSON-LD Streaming Parser	jsonld.js
toRdf-manifest.jsonld (999 triples)	683.964ms (38MB)	708.975ms (40MB)
sparql-init.json (69 triples)	931.698ms (40MB)	1088.607ms (47MB)
person.json (5 triples)	309.419ms (30MB)	313.138ms (41MB)
dbpedia-10000-expanded.json (10,000 triples)	785.557ms (70MB)	202.363ms (62MB)

Tested files:

• toRdf-manifest.jsonld: The JSON-LD toRdf test manifest. A typical JSON-LD file with a single context.
• sparql-init.json: A Comunica configuration file. A JSON-LD file with a large number of complex, nested, and remote contexts.
• person.jsonld: A very small JSON-LD example from the JSON-LD playground.
• dbpedia-10000-expanded.json First 10000 triples of DBpedia in expanded JSON-LD.

Code for measurements

License

This software is written by Ruben Taelman.

This code is released under the MIT license.

Changelog

v2.0.0 - 2020-04-03

Added

• Add JSON-LD 1.1 support:
  • Refactor list handling to allow nested lists in 1.1
  • Handle property-scoped contexts in raw property values
  • Handle property-based indexing with graph containers
  • Handle @type: @id in property-based index containers
  • Handle property-based indexing
  • Allow context nullification in scoped ctx to clear protections
  • Allow protected term overrides in property-scoped contexts
  • Handle out-of-order type-scoped contexts
  • Handle propagation on type-scoped contexts
  • Handle embedded contexts with disabled propagation
  • Handle (in-order) type-scoped contexts
  • Handle @propagate in prop-scoped contexts
  • Handle property-scoped contexts
  • Handle @graph-@id and @graph-@index containers
  • Handle @type on type maps with string values
  • Handle type maps with string values
  • Handle @type: @none
  • Handle @none in containers
  • Ignore @nest properties in the stack
  • Handle graph containers
  • Handle type containers
  • Parse @type: @json as JSON string literals
  • Add option to normalize language tags in 1.1
  • Handle @direction under the different rdfDirection modes
  • Add support for @prefix terms
• Handle JSON-LD 1.1 Streaming Profile:
  • Detect and handle streaming profile in content type header
• Expose fromHttpResponse helper function
• Implement RDF.Sink interface, Closes #50

Changed

• Add streamingProfile flag that is disabled by default
• Align params on validation with strictValues
• Simplify array handling of type keywords
• Refactor architecture to support multiple objects from values
• Only apply language tag lowercasing in 1.0
• Interpret ints >=1e21 as doubles
• Validate language tags
• Ignore instead of error on invalid compact IRIs

<a name="v1.1.2"></a>