pywordsegment
Advanced tools
A fast concatenated-word segmentation library written in Rust, inspired by wordninja and wordsegment. The binding uses pyo3 to interact with the rust package.
pip3 install pywordsegment
import pywordsegment
# The internal UNIGRAMS & BIGRAMS corpuses are lazy initialized
# once per the whole module. Multiple WordSegmenter instances would
# not create new dictionaries.
# Segments a word to its parts
pywordsegment.WordSegmenter.segment(
text="theusashops",
)
# ["the", "usa", "shops"]
# This function checks whether the substring exists as a whole segment
# inside text.
pywordsegment.WordSegmenter.exist_as_segment(
substring="inter",
text="internationalairport",
)
# False
pywordsegment.WordSegmenter.exist_as_segment(
substring="inter",
text="intermilan",
)
# True
Distributed under the MIT License. See LICENSE for more information.
Gal Ben David - gal@intsights.com
Project Link: https://github.com/intsights/pywordsegment
FAQs
Concatenated-word segmentation Python library written in Rust
We found that pywordsegment demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Research
An impersonated golang.org/x/crypto clone exfiltrates passwords, executes a remote shell stager, and delivers a Rekoobe backdoor on Linux.
Security News
npm rolls out a package release cooldown and scalable trusted publishing updates as ecosystem adoption of install safeguards grows.
Security News
AI agents are writing more code than ever, and that's creating new supply chain risks. Feross joins the Risky Business Podcast to break down what that means for open source security.