Socket
Socket
Sign inDemoInstall

github.com/go-json-experiment/json

Package Overview
Dependencies
0
Alerts
File Explorer

Install Socket

Detect and block malicious and high-risk dependencies

Install

    github.com/go-json-experiment/json

Package json implements semantic processing of JSON as specified in RFC 8259. JSON is a simple data interchange format that can represent primitive data types such as booleans, strings, and numbers, in addition to structured data types such as objects and arrays. Marshal and Unmarshal encode and decode Go values to/from JSON text contained within a []byte. MarshalWrite and UnmarshalRead operate on JSON text by writing to or reading from an io.Writer or io.Reader. MarshalEncode and UnmarshalDecode operate on JSON text by encoding to or decoding from a jsontext.Encoder or jsontext.Decoder. Options may be passed to each of the marshal or unmarshal functions to configure the semantic behavior of marshaling and unmarshaling (i.e., alter how JSON data is understood as Go data and vice versa). jsontext.Options may also be passed to the marshal or unmarshal functions to configure the syntactic behavior of encoding or decoding. The data types of JSON are mapped to/from the data types of Go based on the closest logical equivalent between the two type systems. For example, a JSON boolean corresponds with a Go bool, a JSON string corresponds with a Go string, a JSON number corresponds with a Go int, uint or float, a JSON array corresponds with a Go slice or array, and a JSON object corresponds with a Go struct or map. See the documentation on Marshal and Unmarshal for a comprehensive list of how the JSON and Go type systems correspond. Arbitrary Go types can customize their JSON representation by implementing MarshalerV1, MarshalerV2, UnmarshalerV1, or UnmarshalerV2. This provides authors of Go types with control over how their types are serialized as JSON. Alternatively, users can implement functions that match MarshalFuncV1, MarshalFuncV2, UnmarshalFuncV1, or UnmarshalFuncV2 to specify the JSON representation for arbitrary types. This provides callers of JSON functionality with control over how any arbitrary type is serialized as JSON. A Go struct is naturally represented as a JSON object, where each Go struct field corresponds with a JSON object member. When marshaling, all Go struct fields are recursively encoded in depth-first order as JSON object members except those that are ignored or omitted. When unmarshaling, JSON object members are recursively decoded into the corresponding Go struct fields. Object members that do not match any struct fields, also known as “unknown members”, are ignored by default or rejected if RejectUnknownMembers is specified. The representation of each struct field can be customized in the "json" struct field tag, where the tag is a comma separated list of options. As a special case, if the entire tag is `json:"-"`, then the field is ignored with regard to its JSON representation. The first option is the JSON object name override for the Go struct field. If the name is not specified, then the Go struct field name is used as the JSON object name. JSON names containing commas or quotes, or names identical to "" or "-", can be specified using a single-quoted string literal, where the syntax is identical to the Go grammar for a double-quoted string literal, but instead uses single quotes as the delimiters. By default, unmarshaling uses case-sensitive matching to identify the Go struct field associated with a JSON object name. After the name, the following tag options are supported: omitzero: When marshaling, the "omitzero" option specifies that the struct field should be omitted if the field value is zero as determined by the "IsZero() bool" method if present, otherwise based on whether the field is the zero Go value. This option has no effect when unmarshaling. omitempty: When marshaling, the "omitempty" option specifies that the struct field should be omitted if the field value would have been encoded as a JSON null, empty string, empty object, or empty array. This option has no effect when unmarshaling. string: The "string" option specifies that StringifyNumbers be set when marshaling or unmarshaling a struct field value. This causes numeric types to be encoded as a JSON number within a JSON string, and to be decoded from either a JSON number or a JSON string containing a JSON number. This extra level of encoding is often necessary since many JSON parsers cannot precisely represent 64-bit integers. nocase: When unmarshaling, the "nocase" option specifies that if the JSON object name does not exactly match the JSON name for any of the struct fields, then it attempts to match the struct field using a case-insensitive match that also ignores dashes and underscores. If multiple fields match, the first declared field in breadth-first order takes precedence. This takes precedence even if MatchCaseInsensitiveNames is set to false. This cannot be specified together with the "strictcase" option. strictcase: When unmarshaling, the "strictcase" option specifies that the JSON object name must exactly match the JSON name for the struct field. This takes precedence even if MatchCaseInsensitiveNames is set to true. This cannot be specified together with the "nocase" option. inline: The "inline" option specifies that the JSON representable content of this field type is to be promoted as if they were specified in the parent struct. It is the JSON equivalent of Go struct embedding. A Go embedded field is implicitly inlined unless an explicit JSON name is specified. The inlined field must be a Go struct (that does not implement any JSON methods), jsontext.Value, map[string]T, or an unnamed pointer to such types. When marshaling, inlined fields from a pointer type are omitted if it is nil. Inlined fields of type jsontext.Value and map[string]T are called “inlined fallbacks” as they can represent all possible JSON object members not directly handled by the parent struct. Only one inlined fallback field may be specified in a struct, while many non-fallback fields may be specified. This option must not be specified with any other option (including the JSON name). unknown: The "unknown" option is a specialized variant of the inlined fallback to indicate that this Go struct field contains any number of unknown JSON object members. The field type must be a jsontext.Value, map[string]T, or an unnamed pointer to such types. If DiscardUnknownMembers is specified when marshaling, the contents of this field are ignored. If RejectUnknownMembers is specified when unmarshaling, any unknown object members are rejected regardless of whether an inlined fallback with the "unknown" option exists. This option must not be specified with any other option (including the JSON name). format: The "format" option specifies a format flag used to specialize the formatting of the field value. The option is a key-value pair specified as "format:value" where the value must be either a literal consisting of letters and numbers (e.g., "format:RFC3339") or a single-quoted string literal (e.g., "format:'2006-01-02'"). The interpretation of the format flag is determined by the struct field type. The "omitzero" and "omitempty" options are mostly semantically identical. The former is defined in terms of the Go type system, while the latter in terms of the JSON type system. Consequently they behave differently in some circumstances. For example, only a nil slice or map is omitted under "omitzero", while an empty slice or map is omitted under "omitempty" regardless of nilness. The "omitzero" option is useful for types with a well-defined zero value (e.g., net/netip.Addr) or have an IsZero method (e.g., time.Time.IsZero). Every Go struct corresponds to a list of JSON representable fields which is constructed by performing a breadth-first search over all struct fields (excluding unexported or ignored fields), where the search recursively descends into inlined structs. The set of non-inlined fields in a struct must have unique JSON names. If multiple fields all have the same JSON name, then the one at shallowest depth takes precedence and the other fields at deeper depths are excluded from the list of JSON representable fields. If multiple fields at the shallowest depth have the same JSON name, but exactly one is explicitly tagged with a JSON name, then that field takes precedence and all others are excluded from the list. This is analogous to Go visibility rules for struct field selection with embedded struct types. Marshaling or unmarshaling a non-empty struct without any JSON representable fields results in a SemanticError. Unexported fields must not have any `json` tags except for `json:"-"`. Unmarshal matches JSON object names with Go struct fields using a case-sensitive match, but can be configured to use a case-insensitive match with the "nocase" option. This permits unmarshaling from inputs that use naming conventions such as camelCase, snake_case, or kebab-case. By default, JSON object names for Go struct fields are derived from the Go field name, but may be specified in the `json` tag. Due to JSON's heritage in JavaScript, the most common naming convention used for JSON object names is camelCase. The "format" tag option can be used to alter the formatting of certain types. JSON objects can be inlined within a parent object similar to how Go structs can be embedded within a parent struct. The inlining rules are similar to those of Go embedding, but operates upon the JSON namespace. Go struct fields can be omitted from the output depending on either the input Go value or the output JSON encoding of the value. The "omitzero" option omits a field if it is the zero Go value or implements a "IsZero() bool" method that reports true. The "omitempty" option omits a field if it encodes as an empty JSON value, which we define as a JSON null or empty JSON string, object, or array. In many cases, the behavior of "omitzero" and "omitempty" are equivalent. If both provide the desired effect, then using "omitzero" is preferred. The exact order of JSON object can be preserved through the use of a specialized type that implements MarshalerV2 and UnmarshalerV2. Some Go types have a custom JSON representation where the implementation is delegated to some external package. Consequently, the "json" package will not know how to use that external implementation. For example, the google.golang.org/protobuf/encoding/protojson package implements JSON for all google.golang.org/protobuf/proto.Message types. WithMarshalers and WithUnmarshalers can be used to configure "json" and "protojson" to cooperate together. When implementing HTTP endpoints, it is common to be operating with an io.Reader and an io.Writer. The MarshalWrite and UnmarshalRead functions assist in operating on such input/output types. UnmarshalRead reads the entirety of the io.Reader to ensure that io.EOF is encountered without any unexpected bytes after the top-level JSON value. If a type implements encoding.TextMarshaler and/or encoding.TextUnmarshaler, then the MarshalText and UnmarshalText methods are used to encode/decode the value to/from a JSON string. Due to version skew, the set of JSON object members known at compile-time may differ from the set of members encountered at execution-time. As such, it may be useful to have finer grain handling of unknown members. This package supports preserving, rejecting, or discarding such members.


Version published

Readme

Source

JSON Serialization (v2)

GoDev Build Status

This module hosts an experimental implementation of v2 encoding/json. The API is unstable and breaking changes will regularly be made. Do not depend on this in publicly available modules.

Any commits that make breaking API or behavior changes will be marked with the string "WARNING: " near the top of the commit message. It is your responsibility to inspect the list of commit changes when upgrading the module. Not all breaking changes will lead to build failures.

A Discussion about including this package in Go as encoding/json/v2 has been started on the Go Github project on 2023-10-05. Please provide your feedback there.

Goals and objectives

  • Mostly backwards compatible: If possible, v2 should aim to be mostly compatible with v1 in terms of both API and default behavior to ease migration. For example, the Marshal and Unmarshal functions are the most widely used declarations in the v1 package. It seems sensible for equivalent functionality in v2 to be named the same and have a mostly compatible signature. Behaviorally, we should aim for 95% to 99% backwards compatibility. We do not aim for 100% compatibility since we want the freedom to break certain behaviors that are now considered to have been a mistake. We may provide options that can bring the v2 implementation to 100% compatibility, but it will not be the default.

  • More flexible: There is a long list of feature requests. We should aim to provide the most flexible features that addresses most usages. We do not want to over fit the v2 API to handle every possible use case. Ideally, the features provided should be orthogonal in nature such that any combination of features results in as few surprising edge cases as possible.

  • More performant: JSON serialization is widely used and any bit of extra performance gains will be greatly appreciated. Some rarely used behaviors of v1 may be dropped in favor of better performance. For example, despite Encoder and Decoder operating on an io.Writer and io.Reader, they do not operate in a truly streaming manner, leading to a loss in performance. The v2 implementation should aim to be truly streaming by default (see #33714).

  • Easy to use (hard to misuse): The v2 API should aim to make the common case easy and the less common case at least possible. The API should avoid behavior that goes contrary to user expectation, which may result in subtle bugs (see #36225).

  • v1 and v2 maintainability: Since the v1 implementation must stay forever, it would be beneficial if v1 could be implemented under the hood with v2, allowing for less maintenance burden in the future. This probably implies that behavioral changes in v2 relative to v1 need to be exposed as options.

  • Avoid unsafe: Standard library packages generally avoid the use of package unsafe even if it could provide a performance boost. We aim to preserve this property.

Expectations

While this module aims to possibly be the v2 implementation of encoding/json, there is no guarantee that this outcome will occur. As with any major change to the Go standard library, this will eventually go through the Go proposal process. At the present moment, this is still in the design and experimentation phase and is not ready for a formal proposal.

There are several possible outcomes from this experiment:

  1. We determine that a v2 encoding/json would not provide sufficient benefit over the existing v1 encoding/json package. Thus, we abandon this effort.
  2. We propose a v2 encoding/json design, but it is rejected in favor of some other design that is considered superior.
  3. We propose a v2 encoding/json design, but rather than adding an entirely new v2 encoding/json package, we decide to merge its functionality into the existing v1 encoding/json package.
  4. We propose a v2 encoding/json design and it is accepted, resulting in its addition to the standard library.
  5. Some other unforeseen outcome (among the infinite number of possibilities).

Development

This module is primarily developed by @dsnet, @mvdan, and @johanbrandhorst with feedback provided by @rogpeppe, @ChrisHines, and @rsc.

Discussion about semantics occur semi-regularly, where a record of past meetings can be found here.

Design overview

This package aims to provide a clean separation between syntax and semantics. Syntax deals with the structural representation of JSON (as specified in RFC 4627, RFC 7159, RFC 7493, RFC 8259, and RFC 8785). Semantics deals with the meaning of syntactic data as usable application data.

The Encoder and Decoder types are streaming tokenizers concerned with the packing or parsing of JSON data. They operate on Token and Value types which represent the common data structures that are representable in JSON. Encoder and Decoder do not aim to provide any interpretation of the data.

Functions like Marshal, MarshalWrite, MarshalEncode, Unmarshal, UnmarshalRead, and UnmarshalDecode provide semantic meaning by correlating any arbitrary Go type with some JSON representation of that type (as stored in data types like []byte, io.Writer, io.Reader, Encoder, or Decoder).

API overview

This diagram provides a high-level overview of the v2 json and jsontext packages. Purple blocks represent types, while blue blocks represent functions or methods. The arrows and their direction represent the approximate flow of data. The bottom half of the diagram contains functionality that is only concerned with syntax (implemented by the jsontext package), while the upper half contains functionality that assigns semantic meaning to syntactic data handled by the bottom half (as implemented by the v2 json package).

In contrast to v1 encoding/json, options are represented as separate types rather than being setter methods on the Encoder or Decoder types. Some options affects JSON serialization at the syntactic layer, while others affect it at the semantic layer. Some options only affect JSON when decoding, while others affect JSON while encoding.

Behavior changes

The v2 json package changes the default behavior of Marshal and Unmarshal relative to the v1 json package to be more sensible. Some of these behavior changes have options and workarounds to opt into behavior similar to what v1 provided.

This table shows an overview of the changes:

v1v2Details
JSON object members are unmarshaled into a Go struct using a case-insensitive name match.JSON object members are unmarshaled into a Go struct using a case-sensitive name match.CaseSensitivity
When marshaling a Go struct, a struct field marked as omitempty is omitted if the field value is an empty Go value, which is defined as false, 0, a nil pointer, a nil interface value, and any empty array, slice, map, or string.When marshaling a Go struct, a struct field marked as omitempty is omitted if the field value would encode as an empty JSON value, which is defined as a JSON null, or an empty JSON string, object, or array.OmitEmptyOption
The string option does affect Go bools.The string option does not affect Go bools.StringOption
The string option does not recursively affect sub-values of the Go field value.The string option does recursively affect sub-values of the Go field value.StringOption
The string option sometimes accepts a JSON null escaped within a JSON string.The string option never accepts a JSON null escaped within a JSON string.StringOption
A nil Go slice is marshaled as a JSON null.A nil Go slice is marshaled as an empty JSON array.NilSlicesAndMaps
A nil Go map is marshaled as a JSON null.A nil Go map is marshaled as an empty JSON object.NilSlicesAndMaps
A Go array may be unmarshaled from a JSON array of any length.A Go array must be unmarshaled from a JSON array of the same length.Arrays
A Go byte array is represented as a JSON array of JSON numbers.A Go byte array is represented as a Base64-encoded JSON string.ByteArrays
MarshalJSON and UnmarshalJSON methods declared on a pointer receiver are inconsistently called.MarshalJSON and UnmarshalJSON methods declared on a pointer receiver are consistently called.PointerReceiver
A Go map is marshaled in a deterministic order.A Go map is marshaled in a non-deterministic order.MapDeterminism
JSON strings are encoded with HTML-specific characters being escaped.JSON strings are encoded without any characters being escaped (unless necessary).EscapeHTML
When marshaling, invalid UTF-8 within a Go string are silently replaced.When marshaling, invalid UTF-8 within a Go string results in an error.InvalidUTF8
When unmarshaling, invalid UTF-8 within a JSON string are silently replaced.When unmarshaling, invalid UTF-8 within a JSON string results in an error.InvalidUTF8
When marshaling, an error does not occur if the output JSON value contains objects with duplicate names.When marshaling, an error does occur if the output JSON value contains objects with duplicate names.DuplicateNames
When unmarshaling, an error does not occur if the input JSON value contains objects with duplicate names.When unmarshaling, an error does occur if the input JSON value contains objects with duplicate names.DuplicateNames
Unmarshaling a JSON null into a non-empty Go value inconsistently clears the value or does nothing.Unmarshaling a JSON null into a non-empty Go value always clears the value.MergeNull
Unmarshaling a JSON value into a non-empty Go value follows inconsistent and bizarre behavior.Unmarshaling a JSON value into a non-empty Go value always merges if the input is an object, and otherwise replaces.MergeComposite
A time.Duration is represented as a JSON number containing the decimal number of nanoseconds.A time.Duration is represented as a JSON string containing the formatted duration (e.g., "1h2m3.456s").TimeDurations
Unmarshaling a JSON number into a Go float beyond its representation results in an error.Unmarshaling a JSON number into a Go float beyond its representation uses the closest representable value (e.g., ±math.MaxFloat).MaxFloats
A Go struct with only unexported fields can be serialized.A Go struct with only unexported fields cannot be serialized.EmptyStructs
A Go struct that embeds an unexported struct type can sometimes be serialized.A Go struct that embeds an unexported struct type cannot be serialized.EmbedUnexported

See diff_test.go for details about every change.

Performance

One of the goals of the v2 module is to be more performant than v1, but not at the expense of correctness. In general, v2 is at performance parity with v1 for marshaling, but dramatically faster for unmarshaling.

See https://github.com/go-json-experiment/jsonbench for benchmarks comparing v2 with v1 and a number of other popular JSON implementations.

FAQs

Last updated on 02 Nov 2023

Did you know?

Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.

Install

Related posts

SocketSocket SOC 2 Logo

Product

  • Package Alerts
  • Integrations
  • Docs
  • Pricing
  • FAQ
  • Roadmap

Stay in touch

Get open source security insights delivered straight into your inbox.


  • Terms
  • Privacy
  • Security

Made with ⚡️ by Socket Inc