@exodus/bytes

@exodus/bytes

Uint8Array conversion to and from base64, base32, base58, hex, utf8, utf16, bech32 and wif

And a TextEncoder / TextDecoder polyfill

Strict

Performs proper input validation, ensures no garbage-in-garbage-out

Tested on Node.js, Deno, Bun, browsers (including Servo), Hermes, QuickJS and barebone engines in CI (how?)

Fast

• 10-20x faster than Buffer polyfill
• 2-10x faster than iconv-lite

The above was for the js fallback

It's up to 100x when native impl is available
e.g. in utf8fromString on Hermes / React Native or fromHex in Chrome

Also:

• 3-8x faster than bs58
• 10-30x faster than @scure/base (or >100x on Node.js <25)
• Faster in utf8toString / utf8fromString than Buffer or TextDecoder / TextEncoder on Node.js

See Performance for more info

TextEncoder / TextDecoder polyfill

import { TextDecoder, TextEncoder } from '@exodus/bytes/encoding.js'
import { TextDecoderStream, TextEncoderStream } from '@exodus/bytes/encoding.js' // Requires Streams

Less than half the bundle size of text-encoding, whatwg-encoding or iconv-lite (gzipped or not).
Also much faster than all of those.

[!TIP] See also the lite version to get this down to 10 KiB gzipped.

Spec compliant, passing WPT and covered with extra tests.
Moreover, tests for this library uncovered bugs in all major implementations.
Including all three major browser engines being wrong at UTF-8.
See WPT pull request.

It works correctly even in environments that have native implementations broken (that's all of them currently).
Runs (and passes WPT) on Node.js built without ICU.

[!NOTE] Faster than Node.js native implementation on Node.js.

The JS multi-byte version is as fast as native impl in Node.js and browsers, but (unlike them) returns correct results.

For encodings where native version is known to be fast and correct, it is automatically used.
Some single-byte encodings are faster than native in all three major browser engines.

See analysis table for more info.

Caveat: TextDecoder / TextEncoder APIs are lossy by default per spec

These are only provided as a compatibility layer, prefer hardened APIs instead in new code.

• TextDecoder can (and should) be used with { fatal: true } option for all purposes demanding correctness / lossless transforms

• TextEncoder does not support a fatal mode per spec, it always performs replacement.

  That is not suitable for hashing, cryptography or consensus applications.
  Otherwise there would be non-equal strings with equal signatures and hashes — the collision is caused by the lossy transform of a JS string to bytes. Those also survive e.g. JSON.stringify/JSON.parse or being sent over network.

  Use strict APIs in new applications, see utf8fromString / utf16fromString below.
  Those throw on non-well-formed strings by default.

Lite version

If you don't need support for legacy multi-byte encodings, you can use the lite import:

import { TextDecoder, TextEncoder } from '@exodus/bytes/encoding-lite.js'
import { TextDecoderStream, TextEncoderStream } from '@exodus/bytes/encoding-lite.js' // Requires Streams

This reduces the bundle size 9x:
from 90 KiB gzipped for @exodus/bytes/encoding.js to 10 KiB gzipped for @exodus/bytes/encoding-lite.js.
(For comparison, text-encoding module is 190 KiB gzipped, and iconv-lite is 194 KiB gzipped):

It still supports utf-8, utf-16le, utf-16be and all single-byte encodings specified by the spec, the only difference is support for legacy multi-byte encodings.

See the list of encodings.

API

@exodus/bytes/utf8.js

import { utf8fromString, utf8toString } from '@exodus/bytes/utf8.js'

// loose
import { utf8fromStringLoose, utf8toStringLoose } from '@exodus/bytes/utf8.js'

utf8fromString(str, format = 'uint8')

utf8fromStringLoose(str, format = 'uint8')

utf8toString(arr)

utf8toStringLoose(arr)

@exodus/bytes/utf16.js

import { utf16fromString, utf16toString } from '@exodus/bytes/utf16.js'

// loose
import { utf16fromStringLoose, utf16toStringLoose } from '@exodus/bytes/utf16.js'

utf16fromString(str, format = 'uint16')

utf16fromStringLoose(str, format = 'uint16')

utf16toString(arr, 'uint16')

utf16toStringLoose(arr, 'uint16')

@exodus/bytes/single-byte.js

import { createSinglebyteDecoder, createSinglebyteEncoder } from '@exodus/bytes/single-byte.js'
import { windows1252toString, windows1252fromString } from '@exodus/bytes/single-byte.js'
import { latin1toString, latin1fromString } from '@exodus/bytes/single-byte.js'

Decode / encode the legacy single-byte encodings according to the Encoding standard (§9, §14.5), and unicode.org iso-8859-* mappings.

Supports all single-byte encodings listed in the WHATWG Encoding standard: ibm866, iso-8859-2, iso-8859-3, iso-8859-4, iso-8859-5, iso-8859-6, iso-8859-7, iso-8859-8, iso-8859-8-i, iso-8859-10, iso-8859-13, iso-8859-14, iso-8859-15, iso-8859-16, koi8-r, koi8-u, macintosh, windows-874, windows-1250, windows-1251, windows-1252, windows-1253, windows-1254, windows-1255, windows-1256, windows-1257, windows-1258, x-mac-cyrillic and x-user-defined.

Also supports iso-8859-1, iso-8859-9, iso-8859-11 as defined at unicode.org (and all other iso-8859-* encodings there as they match WHATWG).

[!NOTE] While all iso-8859-* encodings supported by the WHATWG Encoding standard match unicode.org, the WHATWG Encoding spec doesn't support iso-8859-1, iso-8859-9, iso-8859-11, and instead maps them as labels to windows-1252, windows-1254, windows-874.
createSinglebyteDecoder() (unlike TextDecoder or legacyHookDecode()) does not do such mapping, so its results will differ from TextDecoder for those encoding names.

> new TextDecoder('iso-8859-1').encoding
'windows-1252'
> new TextDecoder('iso-8859-9').encoding
'windows-1254'
> new TextDecoder('iso-8859-11').encoding
'windows-874'
> new TextDecoder('iso-8859-9').decode(Uint8Array.of(0x80, 0x81, 0xd0))
'€\x81Ğ' // this is actually decoded according to windows-1254 per TextDecoder spec
> createSinglebyteDecoder('iso-8859-9')(Uint8Array.of(0x80, 0x81, 0xd0))
'\x80\x81Ğ' // this is iso-8859-9 as defined at https://unicode.org/Public/MAPPINGS/ISO8859/8859-9.txt

All WHATWG Encoding spec windows-* encodings are supersets of corresponding unicode.org encodings, meaning that they encode/decode all the old valid (non-replacement) strings / byte sequences identically, but can also support a wider range of inputs.

createSinglebyteDecoder(encoding, loose = false)

Create a decoder for a supported one-byte encoding, given its lowercased name encoding.

Returns a function decode(arr) that decodes bytes to a string.

createSinglebyteEncoder(encoding, { mode = 'fatal' })

Create an encoder for a supported one-byte encoding, given its lowercased name encoding.

Returns a function encode(string) that encodes a string to bytes.

In 'fatal' mode (default), will throw on non well-formed strings or any codepoints which could not be encoded in the target encoding.

latin1toString(arr)

Decode iso-8859-1 bytes to a string.

There is no loose variant for this encoding, all bytes can be decoded.

Same as:

const latin1toString = createSinglebyteDecoder('iso-8859-1')

Note: this is different from new TextDecoder('iso-8859-1') and new TextDecoder('latin1'), as those alias to new TextDecoder('windows-1252').

latin1fromString(string)

Encode a string to iso-8859-1 bytes.

Will throw on non well-formed strings or any codepoints which could not be encoded in iso-8859-1.

Same as:

const latin1fromString = createSinglebyteEncoder('iso-8859-1', { mode: 'fatal' })

windows1252toString(arr)

Decode windows-1252 bytes to a string.

There is no loose variant for this encoding, all bytes can be decoded.

Same as:

const windows1252toString = createSinglebyteDecoder('windows-1252')

windows1252fromString(string)

Encode a string to windows-1252 bytes.

Will throw on non well-formed strings or any codepoints which could not be encoded in windows-1252.

Same as:

const windows1252fromString = createSinglebyteEncoder('windows-1252', { mode: 'fatal' })

@exodus/bytes/multi-byte.js

import { createMultibyteDecoder } from '@exodus/bytes/multi-byte.js'

Decode the legacy multi-byte encodings according to the Encoding standard (§10, §11, §12, §13).

Supports all legacy multi-byte encodings listed in the standard: gbk, gb18030, big5, euc-jp, iso-2022-jp, shift_jis, euc-kr.

createMultibyteDecoder(encoding, loose = false)

Create a decoder for a supported legacy multi-byte encoding, given its lowercased name encoding.

Returns a function decode(arr, stream = false) that decodes bytes to a string.

That function will have state while stream = true is used.

@exodus/bytes/bigint.js

import { fromBigInt, toBigInt } from '@exodus/bytes/bigint.js'

fromBigInt(bigint, { length, format = 'uint8' })

toBigInt(arr)

@exodus/bytes/hex.js

Implements Base16 from RFC4648 (no differences from RFC3548).

import { fromHex, toHex } from '@exodus/bytes/hex.js'

fromHex(string)

toHex(arr)

@exodus/bytes/base64.js

Implements Base64 from RFC4648 (no differences from RFC3548).

import { fromBase64, toBase64 } from '@exodus/bytes/base64.js'
import { fromBase64url, toBase64url } from '@exodus/bytes/base64.js'
import { fromBase64any } from '@exodus/bytes/base64.js'

fromBase64(str, { format = 'uint8', padding = 'both' })

fromBase64url(str, { format = 'uint8', padding = false })

fromBase64any(str, { format = 'uint8', padding = 'both' })

toBase64(arr, { padding = true })

toBase64url(arr, { padding = false })

@exodus/bytes/base32.js

Implements Base32 from RFC4648 (no differences from RFC3548).

import { fromBase32, toBase32 } from '@exodus/bytes/base32.js'
import { fromBase32hex, toBase32hex } from '@exodus/bytes/base32.js'

fromBase32(str, { format = 'uint8', padding = 'both' })

fromBase32hex(str, { format = 'uint8', padding = 'both' })

toBase32(arr, { padding = false })

toBase32hex(arr, { padding = false })

@exodus/bytes/bech32.js

Implements BIP-0173 and BIP-0350.

import { fromBech32, toBech32 } from '@exodus/bytes/bech32.js'
import { fromBech32m, toBech32m } from '@exodus/bytes/bech32.js'
import { getPrefix } from '@exodus/bytes/bech32.js'

getPrefix(str, limit = 90)

fromBech32(str, limit = 90)

toBech32(prefix, bytes, limit = 90)

fromBech32m(str, limit = 90)

toBech32m(prefix, bytes, limit = 90)

@exodus/bytes/base58.js

import { fromBase58, toBase58 } from '@exodus/bytes/base58.js'
import { fromBase58xrp, toBase58xrp } from '@exodus/bytes/base58.js'

fromBase58(str, format = 'uint8')

toBase58(arr)

fromBase58xrp(str, format = 'uint8')

toBase58xrp(arr)

@exodus/bytes/base58check.js

import { fromBase58check, toBase58check } from '@exodus/bytes/base58check.js'
import { fromBase58checkSync, toBase58checkSync } from '@exodus/bytes/base58check.js'
import { makeBase58check } from '@exodus/bytes/base58check.js'

On non-Node.js, requires peer dependency @noble/hashes to be installed.

async fromBase58check(str, format = 'uint8')

async toBase58check(arr)

fromBase58checkSync(str, format = 'uint8')

toBase58checkSync(arr)

makeBase58check(hashAlgo, hashAlgoSync)

@exodus/bytes/wif.js

import { fromWifString, toWifString } from '@exodus/bytes/wif.js'
import { fromWifStringSync, toWifStringSync } from '@exodus/bytes/wif.js'

On non-Node.js, requires peer dependency @noble/hashes to be installed.

async fromWifString(string, version)

fromWifStringSync(string, version)

async toWifString({ version, privateKey, compressed })

toWifStringSync({ version, privateKey, compressed })

@exodus/bytes/encoding.js

import { TextDecoder, TextEncoder } from '@exodus/bytes/encoding.js'
import { TextDecoderStream, TextEncoderStream } from '@exodus/bytes/encoding.js' // Requires Streams

// Hooks for standards
import { getBOMEncoding, legacyHookDecode, labelToName, normalizeEncoding } from '@exodus/bytes/encoding.js'

Implements the Encoding standard: TextDecoder, TextEncoder, TextDecoderStream, TextEncoderStream, some hooks (see below).

new TextDecoder(label = 'utf-8', { fatal = false, ignoreBOM = false })

TextDecoder implementation/polyfill.

new TextEncoder()

TextEncoder implementation/polyfill.

new TextDecoderStream(label = 'utf-8', { fatal = false, ignoreBOM = false })

TextDecoderStream implementation/polyfill.

Requires Streams to be either supported by the platform or polyfilled.

new TextEncoderStream()

TextEncoderStream implementation/polyfill.

Requires Streams to be either supported by the platform or polyfilled.

labelToName(label)

Implements get an encoding from a string label.

Converts an encoding label to its name, as a case-sensitive string.

If an encoding with that label does not exist, returns null.

All encoding names are also valid labels for corresponding encodings.

normalizeEncoding(label)

Converts an encoding label to its name, as an ASCII-lowercased string.

If an encoding with that label does not exist, returns null.

This is the same as decoder.encoding getter, except that it:

• Supports replacement encoding and its labels
• Does not throw for invalid labels and instead returns null

It is identical to:

labelToName(label)?.toLowerCase() ?? null

All encoding names are also valid labels for corresponding encodings.

getBOMEncoding(input)

Implements BOM sniff legacy hook.

Given a TypedArray or an ArrayBuffer instance input, returns either of:

• 'utf-8', if input starts with UTF-8 byte order mark.
• 'utf-16le', if input starts with UTF-16LE byte order mark.
• 'utf-16be', if input starts with UTF-16BE byte order mark.
• null otherwise.

legacyHookDecode(input, fallbackEncoding = 'utf-8')

Implements decode legacy hook.

Given a TypedArray or an ArrayBuffer instance input and an optional fallbackEncoding encoding label, sniffs encoding from BOM with fallbackEncoding fallback and then decodes the input using that encoding, skipping BOM if it was present.

Notes:

• BOM-sniffed encoding takes precedence over fallbackEncoding option per spec. Use with care.
• Always operates in non-fatal mode, aka replacement. It can convert different byte sequences to equal strings.

This method is similar to the following code, except that it doesn't support encoding labels and only expects lowercased encoding name:

new TextDecoder(getBOMEncoding(input) ?? fallbackEncoding).decode(input)

@exodus/bytes/encoding-lite.js

import { TextDecoder, TextEncoder } from '@exodus/bytes/encoding-lite.js'
import { TextDecoderStream, TextEncoderStream } from '@exodus/bytes/encoding-lite.js' // Requires Streams

// Hooks for standards
import { getBOMEncoding, legacyHookDecode, labelToName, normalizeEncoding } from '@exodus/bytes/encoding-lite.js'

The exact same exports as @exodus/bytes/encoding.js are also exported as @exodus/bytes/encoding-lite.js, with the difference that the lite version does not load multi-byte TextDecoder encodings by default to reduce bundle size 10x.

The only affected encodings are: gbk, gb18030, big5, euc-jp, iso-2022-jp, shift_jis and their labels when used with TextDecoder.

Legacy single-byte encodingds are loaded by default in both cases.

TextEncoder and hooks for standards (including labelToName / normalizeEncoding) do not have any behavior differences in the lite version and support full range if inputs.

To avoid inconsistencies, the exported classes and methods are exactly the same objects.

> lite = require('@exodus/bytes/encoding-lite.js')
[Module: null prototype] {
  TextDecoder: [class TextDecoder],
  TextDecoderStream: [class TextDecoderStream],
  TextEncoder: [class TextEncoder],
  TextEncoderStream: [class TextEncoderStream],
  getBOMEncoding: [Function: getBOMEncoding],
  labelToName: [Function: labelToName],
  legacyHookDecode: [Function: legacyHookDecode],
  normalizeEncoding: [Function: normalizeEncoding]
}
> new lite.TextDecoder('big5').decode(Uint8Array.of(0x25))
Uncaught:
Error: Legacy multi-byte encodings are disabled in /encoding-lite.js, use /encoding.js for full encodings range support

> full = require('@exodus/bytes/encoding.js')
[Module: null prototype] {
  TextDecoder: [class TextDecoder],
  TextDecoderStream: [class TextDecoderStream],
  TextEncoder: [class TextEncoder],
  TextEncoderStream: [class TextEncoderStream],
  getBOMEncoding: [Function: getBOMEncoding],
  labelToName: [Function: labelToName],
  legacyHookDecode: [Function: legacyHookDecode],
  normalizeEncoding: [Function: normalizeEncoding]
}
> full.TextDecoder === lite.TextDecoder
true
> new full.TextDecoder('big5').decode(Uint8Array.of(0x25))
'%'
> new lite.TextDecoder('big5').decode(Uint8Array.of(0x25))
'%'

@exodus/bytes/encoding-browser.js

import { TextDecoder, TextEncoder } from '@exodus/bytes/encoding-browser.js'
import { TextDecoderStream, TextEncoderStream } from '@exodus/bytes/encoding-browser.js' // Requires Streams

// Hooks for standards
import { getBOMEncoding, legacyHookDecode, labelToName, normalizeEncoding } from '@exodus/bytes/encoding-browser.js'

Same as @exodus/bytes/encoding.js, but in browsers instead of polyfilling just uses whatever the browser provides, drastically reducing the bundle size (to less than 2 KiB gzipped).

Under non-browser engines (Node.js, React Native, etc.) a full polyfill is used as those platforms do not provide sufficiently complete / non-buggy TextDecoder APIs.

[!NOTE] Implementations in browsers have bugs, but they are fixing them and the expected update window is short.
If you want to circumvent browser bugs, use full @exodus/bytes/encoding.js import.

License

MIT