Socket
Socket
Sign inDemoInstall

pyunormalize

Package Overview
Dependencies
0
Maintainers
1
Alerts
File Explorer

Install Socket

Protect your apps from supply chain attacks

Install

pyunormalize

Unicode normalization forms (NFC, NFKC, NFD, NFKD). A library independent from the Python core Unicode database.

    15.1.0

Maintainers
1

Readme

pyunormalize

A pure Python implementation of the Unicode normalization algorithm independent from the Python core Unicode database. This package supports version 15.1 of the Unicode standard (released in September 2023). It has been thoroughly tested against the Unicode test file.

For the formal specification of the Unicode normalization algorithm, see Section 3.11, Normalization Forms, in the Unicode core specification.

Installation

The easiest method to install is using pip:

pip install pyunormalize

UCD version

To get the version of the Unicode character database currently used:

>>> from pyunormalize import UCD_VERSION
>>> UCD_VERSION
'15.1.0'

Example usage

>>> from pyunormalize import NFC, NFD, NFKC, NFKD
>>> s = "élève"  # "\u00E9\u006C\u00E8\u0076\u0065"
>>> nfc = NFC(s)
>>> nfd = NFD(s)
>>> nfc == s
True
>>> nfd == nfc
False
>>> " ".join([f"{ord(x):04X}" for x in nfc])
'00E9 006C 00E8 0076 0065'
>>> " ".join([f"{ord(x):04X}" for x in nfd])
'0065 0301 006C 0065 0300 0076 0065'
>>>
>>> s = "⑴ ffi ²"
>>> NFC(s), NFKC(s), NFD(s), NFKD(s)
('⑴ ffi ²', '(1) ffi 2', '⑴ ffi ²', '(1) ffi 2')

>>> from pyunormalize import normalize
>>> normalize("NFKD", "⑴ ffi ²")
'(1) ffi 2'
>>> forms = ["NFC", "NFD", "NFKC", "NFKD"]
>>> [normalize(f, "\u017F\u0307\u0323") for f in forms]
['ẛ̣', 'ẛ̣', 'ṩ', 'ṩ']

This implementation is based on the following resources:

Licenses

The code is available under the MIT license.

Usage of Unicode data files is governed by the UNICODE TERMS OF USE. Further specifications of rights and restrictions pertaining to the use of the Unicode data files and software can be found in the Unicode Data Files and Software License, a copy of which is included as UNICODE-LICENSE.

Keywords

FAQs


Did you know?

Socket installs a GitHub app to automatically flag issues on every pull request and report the health of your dependencies. Find out what is inside your node modules and prevent malicious activity before you update the dependencies.

Install

Related posts

SocketSocket SOC 2 Logo

Product

  • Package Alerts
  • Integrations
  • Docs
  • Pricing
  • FAQ
  • Roadmap

Stay in touch

Get open source security insights delivered straight into your inbox.


  • Terms
  • Privacy
  • Security

Made with ⚡️ by Socket Inc