icu_normalizer

icu_normalizer

Normalizing text into Unicode Normalization Forms.

This module is published as its own crate (icu_normalizer) and as part of the icu crate. See the latter for more details on the ICU4X project.

Functionality

The top level of the crate provides normalization of input into the four normalization forms defined in UAX #15: Unicode Normalization Forms: NFC, NFD, NFKC, and NFKD.

Three kinds of contiguous inputs are supported: known-well-formed UTF-8 (&str), potentially-not-well-formed UTF-8, and potentially-not-well-formed UTF-16. Additionally, an iterator over char can be wrapped in a normalizing iterator.

The uts46 module provides the combination of mapping and normalization operations for UTS #46: Unicode IDNA Compatibility Processing. This functionality is not meant to be used by applications directly. Instead, it is meant as a building block for a full implementation of UTS #46, such as the idna crate.

The properties module provides the non-recursive canonical decomposition operation on a per char basis and the canonical compositon operation given two chars. It also provides access to the Canonical Combining Class property. These operations are primarily meant for HarfBuzz, the types CanonicalComposition, CanonicalDecomposition, and CanonicalCombiningClassMap implement the [harfbuzz_traits] if the harfbuzz_traits Cargo feature is enabled.

Notably, this normalizer does not provide the normalization “quick check” that can result in “maybe” in addition to “yes” and “no”. The normalization checks provided by this crate always give a definitive non-“maybe” answer.

Examples

let nfc = icu_normalizer::ComposingNormalizerBorrowed::new_nfc();
assert_eq!(nfc.normalize("a\u{0308}"), "ä");
assert!(nfc.is_normalized("ä"));

let nfd = icu_normalizer::DecomposingNormalizerBorrowed::new_nfd();
assert_eq!(nfd.normalize("ä"), "a\u{0308}");
assert!(!nfd.is_normalized("ä"));

More Information

For more information on development, authorship, contributing etc. please visit ICU4X home page.