unicode-trie

What is unicode-trie?

The unicode-trie npm package provides a compact and efficient way to store and retrieve Unicode character data using a trie data structure. This is particularly useful for applications that need to handle large sets of Unicode data, such as text processing, font rendering, and internationalization.

What are unicode-trie's main functionalities?

Creating a Unicode Trie

This feature allows you to create a Unicode trie by setting values for specific Unicode code points. The `UnicodeTrieBuilder` is used to build the trie, and the `freeze` method finalizes it.

const UnicodeTrieBuilder = require('unicode-trie/builder');
const builder = new UnicodeTrieBuilder();
builder.set(0x61, 1); // Set value 1 for character 'a'
builder.set(0x62, 2); // Set value 2 for character 'b'
const trie = builder.freeze();

Querying a Unicode Trie

This feature allows you to query a Unicode trie for the value associated with a specific Unicode code point. The `get` method retrieves the value for the given code point.

const UnicodeTrie = require('unicode-trie');
const trie = new UnicodeTrie(buffer); // buffer is the serialized trie data
console.log(trie.get(0x61)); // Get value for character 'a', should output 1
console.log(trie.get(0x62)); // Get value for character 'b', should output 2

Serializing and Deserializing a Unicode Trie

This feature allows you to serialize a Unicode trie to a buffer and then deserialize it back. This is useful for saving the trie to a file or sending it over a network.

const buffer = trie.toBuffer(); // Serialize the trie to a buffer
const deserializedTrie = new UnicodeTrie(buffer); // Deserialize the trie from the buffer

Other packages similar to unicode-trie

unorm

The unorm package provides Unicode normalization forms as specified in Unicode Standard Annex #15. While it focuses on normalization rather than trie-based storage, it is useful for applications that need to handle Unicode text processing.

unicode-properties

The unicode-properties package provides access to Unicode character properties. It is similar to unicode-trie in that it deals with Unicode data, but it focuses on providing property information rather than a trie-based storage mechanism.

unicode-categories

The unicode-categories package provides a way to check Unicode character categories. It is useful for applications that need to classify characters, but it does not offer the trie-based storage and retrieval capabilities of unicode-trie.

README

unicode-trie

A data structure for fast Unicode character metadata lookup, ported from ICU

Background

When implementing many Unicode algorithms such as text segmentation, normalization, bidi processing, etc., fast access to character metadata is crucial to good performance. There over a million code points in the Unicode standard, many of which produce the same result when looked up, so an array or hash table is not appropriate - those data structures are fast but would require a lot of memory. The data is generally grouped in ranges, so you could do a binary search, but that is not fast enough for some applications.

The International Components for Unicode (ICU) project came up with a data structure based on a Trie that provides fast access to Unicode metadata. The range data is precompiled to a serialized and flattened trie, which is then used at runtime to lookup the necessary data. According to my own tests, this is generally at least 50% faster than binary search, with not too much additional memory required.

Installation

npm install unicode-trie

Building a Trie

Unicode Tries are generally precompiled from data in the Unicode database for faster runtime performance. To build a Unicode Trie, use the UnicodeTrieBuilder class.

UnicodeTrieBuilder = require 'unicode-trie/builder'

# create a trie
t = new UnicodeTrieBuilder

# optional parameters for default value, and error value
# if not provided, both are set to 0
t = new UnicodeTrieBuilder 10, 999

# set individual values and ranges
t.set 0x4567, 99
t.setRange 0x40, 0xe7, 0x1234

# you can lookup a value if you like
t.get 0x4567 # => 99

# get a compiled trie (returns a UnicodeTrie object)
trie = t.freeze()

# write compressed trie to a binary file
fs.writeFile 'data.trie', t.toBuffer()

Using a precompiled Trie

Once you've built a precompiled trie, you can load it into the UnicodeTrie class, which is a readonly representation of the trie. From there, you can lookup values.

UnicodeTrie = require 'unicode-trie'
fs = require 'fs'

# load serialized trie from binary file
data = fs.readFileSync 'data.trie'
trie = new UnicodeTrie data

# lookup a value
trie.get 0x4567 # => 99

License

MIT