Huge News!Announcing our $40M Series B led by Abstract Ventures.Learn More
Socket
Sign inDemoInstall
Socket

zemberek-python

Package Overview
Dependencies
Maintainers
2
Alerts
File Explorer

Advanced tools

Socket logo

Install Socket

Detect and block malicious and high-risk dependencies

Install

zemberek-python

Python port of open source text processing library for Turkish, zemberek-nlp

  • 0.2.3
  • PyPI
  • Socket score

Maintainers
2

ZEMBEREK-PYTHON

Python implementation of Natural Language Processing library for Turkish, zemberek-nlp. It is based on zemberek 0.17.1 and is completely written in Python meaning there is no need to setup a Java development environment to run it.

Source Code

https://github.com/Loodos/zemberek-python

Dependencies

  • antlr4-python3-runtime==4.8
  • numpy>=1.19.0

Supported Modules

Currently, following modules are supported.

  • Core (Partially)

  • TurkishMorphology (Partially)

    • Single Word Analysis
    • Diacritics Ignored Analysis
    • Word Generation
    • Sentence Analysis
    • Ambiguity Resolution
  • Tokenization

    • Sentence Boundary Detection
    • Tokenization
  • Normalization (Partially)

    • Spelling Suggestion
    • Noisy Text Normalization

Installation

You can install the package with pip

pip install zemberek-python

Examples

Example usages can be found in examples.py

Notes

There are some minor changes in codes where original contains some Java specific functionality and data structures. We used Python equivalents as much as we could but sometimes we needed to change them. And it affects the performance and accuracy a bit.

In MultiLevelMphf class, in the original Java implementation, there are some integer multiplication operations which I tried to reimplement using vanilla Python 'int', but the results were not the same. Then I tried it with numpy.int32 and numpy.float32, since default java int and float types are 4 byte. The results were the same with Java, however, oftenly these operations produced RuntimeWarning as the multiplication caused overflow. In Java there were no overflow warnings whatsoever. I could not find a reasonable explanation to this situation, nor I could find a better way to implement it. So I suppressed overflow warnings for MultiLevelMphf. Therefore, please be aware that, this is not a healthy behaviour, and you should be careful using this code.

Credits

This project is Python port of zemberek-nlp.

FAQs


Did you know?

Socket

Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.

Install

Related posts

SocketSocket SOC 2 Logo

Product

  • Package Alerts
  • Integrations
  • Docs
  • Pricing
  • FAQ
  • Roadmap
  • Changelog

Packages

npm

Stay in touch

Get open source security insights delivered straight into your inbox.


  • Terms
  • Privacy
  • Security

Made with ⚡️ by Socket Inc