Socket
Socket
Sign inDemoInstall

spacy-langdetect

Package Overview
Dependencies
2
Maintainers
1
Alerts
File Explorer

Install Socket

Protect your apps from supply chain attacks

Install

spacy-langdetect

Fully customizable language detection pipeline for spaCy

    0.1.2

Maintainers
1

Readme

spacy-langdetect

Fully customizable language detection pipeline for spaCy

Installation

pip install spacy-langdetect

NOTE:

Requires spaCy >= 2.0. This dependency is removed in pip install spacy-langdetect so that it can be used with nightly versions also

Basic usage

Out of the box, under the hood it uses langdetect to detect languages on spaCy's Doc and Span objects.

import spacy
from spacy_langdetect import LanguageDetector
nlp = spacy.load("en")
nlp.add_pipe(LanguageDetector(), name="language_detector", last=True)
text = "This is English text. Er lebt mit seinen Eltern und seiner Schwester in Berlin. Yo me divierto todos los días en el parque. Je m'appelle Angélica Summer, j'ai 12 ans et je suis canadienne."
doc = nlp(text)
# document level language detection. Think of it like average language of document!
print(doc._.language)
# sentence level language detection
for i, sent in enumerate(doc.sents):
    print(sent, sent._.language)

Using your own language detector

Suppose you are not happy with the accuracy of the out of the box language detector or you have your own language detector which you want to use with spaCy pipeline. How do you do it? That's where the language_detection_function argument comes in. The function takes in a Spacy Doc or Span object and can return any python object which is stored in doc._.language and span._.language. For example, let's say you want to use googletrans as your language detection module:

import spacy
from spacy.tokens import Doc, Span
from spacy_langdetect import LanguageDetector
# install using pip install googletrans
from googletrans import Translator
nlp = spacy.load("en")

def custom_detection_function(spacy_object):
    # custom detection function should take a Spacy Doc or a
    assert isinstance(spacy_object, Doc) or isinstance(
        spacy_object, Span), "spacy_object must be a spacy Doc or Span object but it is a {}".format(type(spacy_object))
    detection = Translator().detect(spacy_object.text)
    return {'language':detection.lang, 'score':detection.confidence}

nlp.add_pipe(LanguageDetector(language_detection_function=custom_detection_function), name="language_detector", last=True)
text = "This is English text. Er lebt mit seinen Eltern und seiner Schwester in Berlin. Yo me divierto todos los días en el parque. Je m'appelle Angélica Summer, j'ai 12 ans et je suis canadienne."
doc = nlp(text)
# document level language detection. Think of it like average language of document!
print(doc._.language)
# sentence level language detection
for i, sent in enumerate(doc.sents):
    print(sent, sent._.language)

Similarly you can also use pycld2 and other language detectors with spaCy

FAQs


Did you know?

Socket installs a GitHub app to automatically flag issues on every pull request and report the health of your dependencies. Find out what is inside your node modules and prevent malicious activity before you update the dependencies.

Install

Related posts

SocketSocket SOC 2 Logo

Product

  • Package Alerts
  • Integrations
  • Docs
  • Pricing
  • FAQ
  • Roadmap

Stay in touch

Get open source security insights delivered straight into your inbox.


  • Terms
  • Privacy
  • Security

Made with ⚡️ by Socket Inc