github.com/fredbi/core/json

json

This go library processes JSON streams and buffers.

It is intended to provide a low-level toolkit for functionally demanding use-cases involving JSON data.

It brings JSON support for go-openapi packages.

It is not a replacement for the standard library encoding/json. As such, it does not marshal or unmarshal your favorite go struct to or from JSON.

It does not compete with blazingly fast JSON codec libraries, which essentially target the marshal/unmarshal feature.

Instead, it provides structures and types that keep the dynamic nature of JSON, deferring the resolution of values to native go types.

Essentially, it provides ways to work with JSON documents without the need of some pre-specified go data structures.

What's inside?

The module github.com/fredbi/core/json provides the following tools:

• a JSON lexer/parser for buffers or streams (TODO: add support for ld-json)
• a JSON writer to JSON buffers or streams
• a compacted memory store for JSON values
• a Document that holds the hierachical structure of JSON documents
• a dynamic JSON structure that dynamically maps JSON to familiar go types
• types that support JSON tweaks such as undefined or null values.

What is not inside?

• functionality like json.Unmarshal or json.Marshal that shuffles JSON to/from go data structures.
• JSON schema support is provided by github.com/fredbi/core/jsonschema. This library focuses on low-level aspects of JSON only.
• OpenAPI support is provided by github.com/fredbi/core/spec. This is far beyond the scope of mere JSON processing.

How does this lib compare to others?

We designed this library with the goal to serve advanced use cases such as JSON schema analysis and OpenAPI spec handling.

Besides, we wanted the core of go-openapi to avoid external dependencies for this kind of work.

Depending on your goals, you might find libraries that are a better fit for your use case.

• It does not generate code like github.com/mailru/easyjson. However, it shares a lot of ideas with the lexer and writer from easyjson.
• It does not provide an API to extract JSON values directly like github.com/valyala/fastjson or github.com/tidwall/gjson. However, it shares a lot of ideas with fastjson such as reusable values stored in an Arena.
• It does not support mapping to your structs like encoding/json or drop-in replacements for the standard library like github.com/go-ccy/go-json or github.com/jsoniterator/go.
• TODO: compare github.com/go-faster/jx
• TODO: compare github.com/francoispqt/gojay
• TODO: compare github.com/bytedance/sonic (a JIT JSON serializing/deserializing lib)

On the other hand, it provides a few features that you won't find in the above-mentioned libraries:

• Accurate error reporting with context when lexing JSON
• Support for streams as well as buffers
• Support for JSON pointers and JSONPath expressions
• The ability to build JSON programmatically

What is "verbatim"?

A verbatim document reproduces the original JSON unaltered, that is, including non-significant blank space used for indentation. It also leaves escaped unicode sequence unchanged.

This allows to process JSON with the ability to control the original input (as in a text editor).

In contrast, a default document focuses on JSON semantics, i.e. structure and values: non-significant blanks are ignored and the annoying escaped unicode sequences are resolved as soon a possible into their UTF-8 equivalent.

Verbatim lexer, store, document and writer should be reserved to tools like linters, text editor plugins and the like.

What can I do with a JSON "Document"?

Documents are immutable, but cheap to build or amend using shallow clones. This guards against nasty bugs and allows for a concurrent processing of documents.

Documents maintain the original ordering of keys in objects.

• Marshal / Unmarshal to/from JSON bytes
• Encode / Decode to/from a stream of JSON bytes
• Build or clone & amend using the Builder type
• Walk a document using iterators
• TODO: resolve a JSON Pointer within the document
• TODO: Walk a document using jsonpath expressions. See github.com/fredbi/core/json/jsonpath.

Design goals

• accurate representation of JSON semantics¹.
• accurate error reporting ²
• memory-efficient processing of JSON. The toolkit tries its best to keep a low profile in memory ³.
• immutable documents, cheap to clone and build programmatically
• dependency-free packages, besides test dependencies
• extensible interfaces⁴

Non-goals

This library favors accuracy over speed. It may not be the fastest JSON processor available for go. In particular:

• we trade accurate error reporting for speed

At this moment, it is not a goal to support XXL documents outside of main memory, although this could be a contributed extension of the stores.Store interface.

Likewise, it is not part of the current plan to provide writers to specialized backends such as databases or message-oriented middleware. Again, such a feature may be provided in the future as a contributed extension (while keeping our dependency-free goal).

At this moment, the scope is limited to JSON. YAML data, considered as a superset of JSON, is not part of the picture for now.

Use cases

Support go-openapi v2:

• Document: build an implementation of JSON schema. See github.com/fredbi/core/jsonschema.
• Document: build an implementation of the OpenAPI schema. See github.com/fredbi/core/spec.

Besides, here are a few examples of where this toolkit might be useful:

• lexer/writer: JSON linter, text editor plugin
• Document: implement an API that deals dynamically with json
• dynamic JSON: alternative way to implement an API that deals dynamically with json

Footnotes

• the default lexer is strict regarding JSON semantics. JSON types allow to distinguish undefined from null. ↩

• the lexer does its best to report informative error messages, and capture the context of the offending part, including when dealing with streams. ↩

• this leverages the memory pooling utilities exposed by github.com/fredbi/core/pools. By and large, the toolkit favors memory scarcity over CPU, assuming that much of the CPU-intensive processing may be deferred lazily. For instance, th default implementation for stores.Store compresses long strings. ↩

• contrib modules are proposed to host standalone go modules that may bring their own set of dependencies ↩