A Python port of the Apache Lucene ASCII Folding Filter that converts alphabetic, numeric, and symbolic Unicode characters which are not in the first 127 ASCII characters (the ‘Basic Latin’ Unicode block) into ASCII equivalents, if they exist.
A Python port of the Apache Lucene ASCII Folding Filter that converts alphabetic, numeric, and symbolic Unicode characters which are not in the first 127 ASCII characters (the "Basic Latin" Unicode block) into ASCII equivalents, if they exist.
> from fold_to_ascii import fold
> s = u'Astroturf® paté'
> fold(s)
u'Astroturf pate'
> fold(s, u'?')
u'Astroturf? pate'
This library always removes astral characters, even when a replacement character is specified. PRs welcome if you want to fix this.
There are a few Python unidecode libraries out there
(1,
2). They are based on a Perl program
that makes some interesting choices about what to replace, for example, £
(POUND SIGN) is replaced with the string PS. Also, they do not allow
specifying a replacement character to use other than the empty string for
unmapped characters.
$ virtualenv -p python3 ~/.local/venvs/fold_to_ascii
$ source ~/.local/venvs/fold_to_ascii/bin/activate
$ pip install -r requirements.txt
$ make lint && make test
Requires a pypi account.
$ python setup.py sdist bdist_wheel
$ twine upload dist/*
FAQs
A Python port of the Apache Lucene ASCII Folding Filter that converts alphabetic, numeric, and symbolic Unicode characters which are not in the first 127 ASCII characters (the ‘Basic Latin’ Unicode block) into ASCII equivalents, if they exist.
We found that fold-to-ascii demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
PyPI confirms no security flaws were exploited in the Ultralytics supply chain attack and highlights improvements for safer package publishing.
Security News
Research
The Socket Research Team breaks down a malicious wrapper package that uses obfuscation to harvest credentials and exfiltrate sensitive data.
Research
Security News
Attackers used a malicious npm package typosquatting a popular ESLint plugin to steal sensitive data, execute commands, and exploit developer systems.