Product
Introducing License Enforcement in Socket
Ensure open-source compliance with Socket’s License Enforcement Beta. Set up your License Policy and secure your software!
github.com/fxamacker/cbor
This is a generic CBOR encoder and decoder. It can encode integers and floats to their smallest forms (like float16) when values fit. Each release passes 375+ tests and 250+ million execs fuzzing with 1100+ CBOR files.
What is CBOR? CBOR (RFC 7049) is a binary data format inspired by JSON and MessagePack. CBOR is used in IETF Internet Standards such as COSE (RFC 8152) and CWT (RFC 8392 CBOR Web Token). Even WebAuthn uses CBOR.
Why this CBOR library? It doesn't crash and it has well-balanced qualities: small, fast, safe and easy. It also supports "preferred serialization" by encoding integers and floats to their smallest forms when values fit.
Small apps. Same programs are 4-9 MB smaller by switching to this library. No code gen and the only imported pkg is x448/float16 which is maintained by the same team as this library.
Small data. The toarray
, keyasint
, and omitempty
struct tags shrink size of Go structs encoded to CBOR. Integers encode to smallest form that fits. Floats can shrink from float64 -> float32 -> float16 if values can round-trip.
Fast. v1.3 became faster than a well-known library that uses unsafe
optimizations and code gen. Faster libraries will always exist, but speed is only one factor. This library doesn't use unsafe
optimizations or code gen.
Safe and reliable. It prevents crashes on malicious CBOR data by using extensive tests, coverage-guided fuzzing, data validation, and avoiding Go's unsafe
pkg. Nested levels for CBOR arrays, maps, and tags are limited to 32.
Easy and saves time. It has the same API as Go's encoding/json
when possible. Existing structs don't require changes. Go struct tags like `cbor:"name,omitempty"`
and `json:"name,omitempty"`
work as expected.
Predefined configs make it easier to comply with standards like Canonical CBOR, CTAP2 Canonical CBOR, etc.
Custom configs can be created by setting individual options. E.g., EncOptions.NaNConvert can be set to NaNConvertNone, NaNConvert7e00, NaNConvertQuiet, or NaNConvertPreserveSignal.
Struct tags like keyasint
and toarray
make compact CBOR data such as COSE, CWT, and SenML easier to use.
👉 Comparisons • Status • Design Goals • Features • Standards • Fuzzing • Usage • Security Policy • License
Comparisons are between this newer library and a well-known library that had 1,000+ stars before this library was created. Default build settings for each library were used for all comparisons.
This library is safer. Small malicious CBOR messages are rejected quickly before they exhaust system resources.
This library is smaller. Programs like senmlCat can be 4 MB smaller by switching to this library. Programs using more complex CBOR data types can be 9.2 MB smaller.
This library is faster for encoding and decoding CBOR Web Token (CWT). However, speed is only one factor and it can vary depending on data types and sizes. Unlike the other library, this one doesn't use Go's unsafe
package or code gen.
The resource intensive codec.CborHandle
initialization (in the other library) was placed outside the benchmark loop to make sure their library wasn't penalized.
Doing your own comparisons is highly recommended. Use your most common message sizes and data types.
Version 1.x has:
Each commit passes 375+ tests. Each release also passes 250+ million execs in coverage-guided fuzzing using 1,100+ CBOR files (corpus). See Fuzzing and Code Coverage.
Recent activity:
toarray
struct tag to simplify using CBOR arrays.keyasint
struct tag to simplify using CBOR maps with int keys.Coming soon: support for CBOR tags (major type 6). After that, options for handling duplicate map keys.
This library is designed to be a generic CBOR encoder and decoder. It was initially created for a WebAuthn (FIDO2) server library, because existing CBOR libraries (in Go) didn't meet certain criteria in 2019.
This library is designed to be:
encoding/json
plus keyasint
and toarray
struct tags.unsafe
pkg, coverage >95%, coverage-guided fuzzing, and data validation to avoid crashes on malformed or malicious data.Competing factors are balanced:
unsafe
pkg. For speed, use safe optimizations such as caching struct metadata. v1.4 is faster than a well-known library that uses unsafe
and code gen.Avoiding unsafe
package has benefits. The unsafe
package warns:
Packages that import unsafe may be non-portable and are not protected by the Go 1 compatibility guidelines.
All releases prioritize reliability to avoid crashes on decoding malformed CBOR data. See Fuzzing and Coverage.
Features not in Go's standard library are usually not added. However, the toarray
struct tag in ugorji/go was too useful to ignore. It was added in v1.3 when a project mentioned they were using it with CBOR to save disk space.
encoding/json
plus extra struct tags:
cbor.Encoder
writes CBOR to io.Writer
.cbor.Decoder
reads CBOR from io.Reader
.cbor.Marshal
writes CBOR to []byte
.cbor.Unmarshal
reads CBOR from []byte
.toarray
struct tag allows named struct fields for elements of CBOR arrays.keyasint
struct tag allows named struct fields for elements of CBOR maps with int keys.encoding.BinaryMarshaler
and encoding.BinaryUnmarshaler
interfaces.cbor.RawMessage
which can delay CBOR decoding or precompute CBOR encoding.cbor.Marshaler
and cbor.Unmarshaler
interfaces to allow user-defined types to have custom CBOR encoding and decoding.time.Time
as RFC 3339 formatted text string or Unix time.io.LimitReader
can be used to limit sizes.encoding/json
.)Coming soon: support for CBOR tags (major type 6). After that, options for handling duplicate map keys.
This library implements CBOR as specified in RFC 7049 with minor limitations.
For Go integers, encoder always uses "preferred serialization" which encodes their values to the smallest number of bytes.
Encoder has options that can be set individually to create custom configurations. Easy functions are also provided to create and return modifiable configurations (EncOptions):
EncOptions.Sort:
Encoder has 3 types of options for floating-point data: ShortestFloatMode, InfConvertMode, and NaNConvertMode.
EncOptions.ShortestFloat:
With ShortestFloat16, each floating-point value (including subnormals) can encode float64 -> float32 -> float16 when values can round-trip. Conversions for infinity and NaN use InfConvert and NaNConvert settings.
EncOptions.InfConvert:
EncOptions.NaNConvert:
Float16 conversions use x448/float16 maintained by the same team as this library. All 4+ billion possible conversions are verified to be correct in that library.
Decoder checks for all required well-formedness errors, including all "subkinds" of syntax errors and too little data.
After well-formedness is verified, basic validity errors are handled as follows:
When decoding well-formed CBOR arrays and maps, decoder saves the first error it encounters and continues with the next item. Options to handle this differently may be added in the future.
CBOR tags (type 6) is being added in the next release (milestone v2.0) and is coming soon.
Known limitations:
cbor.UnmarshalTypeError
like Go's encoding/json
.Undefined
(0xf7) value decodes to Go's nil
value. Use CBOR Null
(0xf6) to round-trip with Go's nil
.Like Go's encoding/json
, data validation checks the entire message to prevent partially filled (corrupted) data. This library also prevents crashes and resource exhaustion attacks from malicious CBOR data. Use Go's io.LimitReader
when decoding very large data to limit size.
Over 375 tests must pass before tagging a release. They include all RFC 7049 examples, bugs found by fuzzing, 2 maliciously crafted CBOR data, and over 87 tests with malformed data.
Code coverage must not fall below 95% when tagging a release. Code coverage is 97.9% (go test -cover
) for cbor v1.5 which is among the highest for libraries (in Go) of this type.
Coverage-guided fuzzing must pass 250+ million execs before tagging a release. E.g. v1.4 passed 532+ million execs in coverage-guided fuzzing at the time of release and reached 4+ billion execs 18 days later. Fuzzing uses fxamacker/cbor-fuzz. Default corpus has:
Over 1,100 files (corpus) are used for fuzzing because it includes fuzz-generated corpus.
This project uses Semantic Versioning, so the API is always backwards compatible unless the major version number changes.
The API is the same as encoding/json
when possible.
In addition to the API, the keyasint
and toarray
struct tags are worth knowing. They can reduce programming effort, improve system performance, and reduce the size of serialized data.
package cbor // import "github.com/fxamacker/cbor"
func Marshal(v interface{}, encOpts EncOptions) ([]byte, error)
func Unmarshal(data []byte, v interface{}) error
func Valid(data []byte) (rest []byte, err error)
type Decoder struct{ ... }
func NewDecoder(r io.Reader) *Decoder
func (dec *Decoder) Decode(v interface{}) (err error)
func (dec *Decoder) NumBytesRead() int
type EncOptions struct{ ... }
func CTAP2EncOptions() EncOptions
func CanonicalEncOptions() EncOptions
func CoreDetEncOptions() EncOptions
func PreferredUnsortedEncOptions() EncOptions
type Encoder struct{ ... }
func NewEncoder(w io.Writer, encOpts EncOptions) *Encoder
func (enc *Encoder) Encode(v interface{}) error
func (enc *Encoder) StartIndefiniteByteString() error
func (enc *Encoder) StartIndefiniteTextString() error
func (enc *Encoder) StartIndefiniteArray() error
func (enc *Encoder) StartIndefiniteMap() error
func (enc *Encoder) EndIndefinite() error
type InfConvertMode int
const InfConvertFloat16 InfConvertMode = iota ...
type InvalidUnmarshalError struct{ ... }
type Marshaler interface{ ... }
type NaNConvertMode int
const NaNConvert7e00 NaNConvertMode = iota ...
type RawMessage []byte
type SemanticError struct{ ... }
type ShortestFloatMode int
const ShortestFloatNone ShortestFloatMode = iota ...
type SortMode int
const SortNone SortMode = 0 ...
type SyntaxError struct{ ... }
type UnmarshalTypeError struct{ ... }
type Unmarshaler interface{ ... }
type UnsupportedTypeError struct{ ... }
See API docs for more details.
go get github.com/fxamacker/cbor
Released versions benefit from longer fuzz tests.
👉 Use Go's io.LimitReader
when decoding very large data to limit size.
The API is the same as encoding/json
when possible:
The keyasint
and toarray
struct tags make it easy to use compact CBOR message formats. Internet standards often use CBOR arrays and CBOR maps with int keys to save space.
Using named struct fields instead of array elements or maps with int keys makes code more readable and less error prone.
Decoding CWT (CBOR Web Token) using keyasint
and toarray
struct tags:
// Signed CWT is defined in RFC 8392
type signedCWT struct {
_ struct{} `cbor:",toarray"`
Protected []byte
Unprotected coseHeader
Payload []byte
Signature []byte
}
// Part of COSE header definition
type coseHeader struct {
Alg int `cbor:"1,keyasint,omitempty"`
Kid []byte `cbor:"4,keyasint,omitempty"`
IV []byte `cbor:"5,keyasint,omitempty"`
}
// data is []byte containing signed CWT
var v signedCWT
if err := cbor.Unmarshal(data, &v); err != nil {
return err
}
Encoding CWT (CBOR Web Token) using keyasint
and toarray
struct tags:
// Use signedCWT struct defined in "Decoding CWT" example.
var v signedCWT
...
if data, err := cbor.Marshal(v, cbor.EncOptions{}); err != nil {
return err
}
Decoding SenML using keyasint
struct tag:
// RFC 8428 says, "The data is structured as a single array that
// contains a series of SenML Records that can each contain fields"
type SenMLRecord struct {
BaseName string `cbor:"-2,keyasint,omitempty"`
BaseTime float64 `cbor:"-3,keyasint,omitempty"`
BaseUnit string `cbor:"-4,keyasint,omitempty"`
BaseValue float64 `cbor:"-5,keyasint,omitempty"`
BaseSum float64 `cbor:"-6,keyasint,omitempty"`
BaseVersion int `cbor:"-1,keyasint,omitempty"`
Name string `cbor:"0,keyasint,omitempty"`
Unit string `cbor:"1,keyasint,omitempty"`
Value float64 `cbor:"2,keyasint,omitempty"`
ValueS string `cbor:"3,keyasint,omitempty"`
ValueB bool `cbor:"4,keyasint,omitempty"`
ValueD string `cbor:"8,keyasint,omitempty"`
Sum float64 `cbor:"5,keyasint,omitempty"`
Time float64 `cbor:"6,keyasint,omitempty"`
UpdateTime float64 `cbor:"7,keyasint,omitempty"`
}
// data is a []byte containing SenML
var v []SenMLRecord
if err := cbor.Unmarshal(data, &v); err != nil {
return err
}
Encoding SenML using keyasint
struct tag and ShortestFloat16
encoding option:
// use SenMLRecord struct defined in "Decoding SenML" example
var v []SenMLRecord
...
if data, err := cbor.Marshal(v, cbor.EncOptions{ShortestFloat: cbor.ShortestFloat16}); err != nil {
return err
}
Decoding:
// create a decoder
dec := cbor.NewDecoder(reader)
// decode into empty interface
var i interface{}
err = dec.Decode(&i)
// decode into struct
var stru ExampleStruct
err = dec.Decode(&stru)
// decode into map
var m map[string]string
err = dec.Decode(&m)
// decode into primitive
var f float32
err = dec.Decode(&f)
Encoding:
// create an encoder with canonical CBOR encoding enabled
enc := cbor.NewEncoder(writer, cbor.CanonicalEncOptions())
// encode struct
err = enc.Encode(stru)
// encode map
err = enc.Encode(m)
// encode primitive
err = enc.Encode(f)
Encoding indefinite length array:
enc := cbor.NewEncoder(writer, cbor.EncOptions{})
// start indefinite length array encoding
err = enc.StartIndefiniteArray()
// encode array element
err = enc.Encode(1)
// encode array element
err = enc.Encode([]int{2, 3})
// start nested indefinite length array as array element
err = enc.StartIndefiniteArray()
// encode nested array element
err = enc.Encode(4)
// encode nested array element
err = enc.Encode(5)
// end nested indefinite length array
err = enc.EndIndefinite()
// end indefinite length array
err = enc.EndIndefinite()
More examples.
Go structs are faster than maps with string keys:
Go structs with keyasint
struct tag are faster than maps with integer keys:
Go structs with toarray
struct tag are faster than slice:
Doing your own benchmarks is highly recommended. Use your most common message sizes and data types.
See Benchmarks for fxamacker/cbor.
This project has adopted the Contributor Covenant Code of Conduct. Contact faye.github@gmail.com with any questions or comments.
Please refer to How to Contribute.
For v1, security fixes are provided only for the latest released version since the API won't break compatibility.
To report security vulnerabilities, please email faye.github@gmail.com and allow time for the problem to be resolved before reporting it to the public.
Phrases like "no crashes" or "doesn't crash" mean there are no known crash bugs in the latest version based on results of unit tests and coverage-guided fuzzing. It doesn't imply the software is 100% bug-free or 100% invulnerable to all known and unknown attacks.
Please read the license for additional disclaimers and terms.
Copyright (c) Faye Amacker. All rights reserved.
Licensed under the MIT License.
👉 Comparisons • Status • Design Goals • Features • Standards • Fuzzing • Usage • Security Policy • License
FAQs
Unknown package
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Product
Ensure open-source compliance with Socket’s License Enforcement Beta. Set up your License Policy and secure your software!
Product
We're launching a new set of license analysis and compliance features for analyzing, managing, and complying with licenses across a range of supported languages and ecosystems.
Product
We're excited to introduce Socket Optimize, a powerful CLI command to secure open source dependencies with tested, optimized package overrides.