Socket
Socket
Sign inDemoInstall

nlu

Package Overview
Dependencies
5
Maintainers
2
Alerts
File Explorer

Install Socket

Protect your apps from supply chain attacks

Install

nlu

John Snow Labs NLU provides state of the art algorithms for NLP&NLU with 20000+ of pretrained models in 200+ languages. It enables swift and simple development and research with its powerful Pythonic and Keras inspired API. It is powerd by John Snow Labs powerful Spark NLP library.

    5.1.4

Maintainers
2

Readme

NLU: The Power of Spark NLP, the Simplicity of Python

John Snow Labs' NLU is a Python library for applying state-of-the-art text mining, directly on any dataframe, with a single line of code. As a facade of the award-winning Spark NLP library, it comes with 1000+ of pretrained models in 100+, all production-grade, scalable, and trainable, with everything in 1 line of code.

NLU in Action

See how easy it is to use any of the thousands of models in 1 line of code, there are hundreds of tutorials and simple examples you can copy and paste into your projects to achieve State Of The Art easily.

NLU & Streamlit in Action

This 1 line let's you visualize and play with 1000+ SOTA NLU & NLP models in 200 languages

streamlit run https://raw.githubusercontent.com/JohnSnowLabs/nlu/master/examples/streamlit/01_dashboard.py

NLU provides tight and simple integration into Streamlit, which enables building powerful webapps in just 1 line of code which showcase the. View the NLU&Streamlit documentation or NLU & Streamlit examples section. The entire GIF demo and

All NLU resources overview

Take a look at our official NLU page: https://nlu.johnsnowlabs.com/ for user documentation and examples

RessourceDescription
Install NLUJust run pip install nlu pyspark==3.0.2
The NLU NamespaceFind all the names of models you can load with nlu.load()
The nlu.load(<Model>) functionLoad any of the 1000+ models in 1 line
The nlu.load(<Model>).predict(data) functionPredict on Strings, List of Strings, Numpy Arrays, Pandas, Modin and Spark Dataframes
The nlu.load(<train.Model>).fit(data) functionTrain a text classifier for 2-Class, N-Classes Multi-N-Classes, Named-Entitiy-Recognition or Parts of Speech Tagging
The nlu.load(<Model>).viz(data) functionVisualize the results of Word Embedding Similarity Matrix, Named Entity Recognizers, Dependency Trees & Parts of Speech, Entity Resolution,Entity Linking or Entity Status Assertion
The nlu.load(<Model>).viz_streamlit(data) functionDisplay an interactive GUI which lets you explore and test every model and feature in NLU in 1 click.
General ConceptsGeneral concepts in NLU
The latest release notesNewest features added to NLU
Overview NLU 1-liners examplesMost common used models and their results
Overview NLU 1-liners examples for healthcare modelsMost common used healthcare models and their results
Overview of all NLU tutorials and Examples100+ tutorials on how to use NLU on text datasets for various problems and from various sources like Twitter, Chinese News, Crypto News Headlines, Airline Traffic communication, Product review classifier training,
Connect with us on SlackProblems, questions or suggestions? We have a very active and helpful community of over 2000+ AI enthusiasts putting NLU, Spark NLP & Spark OCR to good use
Discussion ForumMore indepth discussion with the community? Post a thread in our discussion Forum
John Snow Labs MediumArticles and Tutorials on the NLU, Spark NLP and Spark OCR
John Snow Labs YoutubeVideos and Tutorials on the NLU, Spark NLP and Spark OCR
NLU WebsiteThe official NLU website
Github IssuesReport a bug

Getting Started with NLU

To get your hands on the power of NLU, you just need to install it via pip and ensure Java 8 is installed and properly configured. Checkout Quickstart for more infos

pip install nlu pyspark==3.0.2

Loading and predicting with any model in 1 line python

import nlu 
nlu.load('sentiment').predict('I love NLU! <3') 

Loading and predicting with multiple models in 1 line

Get 6 different embeddings in 1 line and use them for downstream data science tasks!

nlu.load('bert elmo albert xlnet glove use').predict('I love NLU! <3') 

What kind of models does NLU provide?

NLU provides everything a data scientist might want to wish for in one line of code!

  • NLU provides everything a data scientist might want to wish for in one line of code!
  • 1000 + pre-trained models
  • 100+ of the latest NLP word embeddings ( BERT, ELMO, ALBERT, XLNET, GLOVE, BIOBERT, ELECTRA, COVIDBERT) and different variations of them
  • 50+ of the latest NLP sentence embeddings ( BERT, ELECTRA, USE) and different variations of them
  • 100+ Classifiers (NER, POS, Emotion, Sarcasm, Questions, Spam)
  • 300+ Supported Languages
  • Summarize Text and Answer Questions with T5
  • Labeled and Unlabeled Dependency parsing
  • Various Text Cleaning and Pre-Processing methods like Stemming, Lemmatizing, Normalizing, Filtering, Cleaning pipelines and more

Classifiers trained on many different datasets

Choose the right tool for the right task! Whether you analyze movies or twitter, NLU has the right model for you!

  • trec6 classifier
  • trec10 classifier
  • spam classifier
  • fake news classifier
  • emotion classifier
  • cyberbullying classifier
  • sarcasm classifier
  • sentiment classifier for movies
  • IMDB Movie Sentiment classifier
  • Twitter sentiment classifier
  • NER pretrained on ONTO notes
  • NER trainer on CONLL
  • Language classifier for 20 languages on the wiki 20 lang dataset.

Utilities for the Data Science NLU applications

Working with text data can sometimes be quite a dirty job. NLU helps you keep your hands clean by providing components that take away from data engineering intensive tasks.

  • Datetime Matcher
  • Pattern Matcher
  • Chunk Matcher
  • Phrases Matcher
  • Stopword Cleaners
  • Pattern Cleaners
  • Slang Cleaner

Where can I see all models available in NLU?

For NLU models to load, see the NLU Namespace or the John Snow Labs Modelshub or go straight to the source.

Supported Data Types

  • Pandas DataFrame and Series
  • Spark DataFrames
  • Modin with Ray backend
  • Modin with Dask backend
  • Numpy arrays
  • Strings and lists of strings

Overview of all tutorials using the NLU-Library

In the following tabular, all available tutorials using NLU are listed. These tutorials will help you learn the usage of the NLU library and on how to use it for your own tasks. Some of the tasks NLU does are translating from any language to the english language, lemmatizing, tokenizing, cleaning text from Symbol or unwanted syntax, spellchecking, detecting entities, analyzing sentiments and many more!

{:.table2}

Tutorial DescriptionNLU Spells UsedOpen In ColabDataset and Paper References
Albert Word Embeddings with NLUalbert, sentiment pos albert emotionOpen In ColabAlbert-Paper, Albert on Github, Albert on TensorFlow, T-SNE, T-SNE-Albert, Albert_Embedding
Bert Word Embeddings with NLUbert, pos sentiment emotion bertOpen In ColabBert-Paper, Bert Github, T-SNE, T-SNE-Bert, Bert_Embedding
BIOBERT Word Embeddings with NLUbiobert , sentiment pos biobert emotionOpen In ColabBioBert-Paper, Bert Github , BERT: Deep Bidirectional Transformers, Bert Github, T-SNE, T-SNE-Biobert, Biobert_Embedding
COVIDBERT Word Embeddings with NLUcovidbert, sentiment covidbert posOpen In ColabCovidBert-Paper, Bert Github, T-SNE, T-SNE-CovidBert, Covidbert_Embedding
ELECTRA Word Embeddings with NLUelectra, sentiment pos en.embed.electra emotionOpen In ColabElectra-Paper, T-SNE, T-SNE-Electra, Electra_Embedding
ELMO Word Embeddings with NLUelmo, sentiment pos elmo emotionOpen In ColabELMO-Paper, Elmo-TensorFlow, T-SNE, T-SNE-Elmo, Elmo-Embedding
GLOVE Word Embeddings with NLUglove, sentiment pos glove emotionOpen In ColabGlove-Paper, T-SNE, T-SNE-Glove , Glove_Embedding
XLNET Word Embeddings with NLUxlnet, sentiment pos xlnet emotionOpen In ColabXLNet-Paper, Bert Github, T-SNE, T-SNE-XLNet, Xlnet_Embedding
Multiple Word-Embeddings and Part of Speech in 1 Line of codebert electra elmo glove xlnet albert posOpen In ColabBert-Paper, Albert-Paper, ELMO-Paper, Electra-Paper, XLNet-Paper, Glove-Paper
Normalzing with NLUnormOpen In Colab-
Detect sentences with NLUsentence_detector.deep, sentence_detector.pragmatic, xx.sentence_detectorOpen In ColabSentence Detector
Spellchecking with NLUn.a.n.a.-
Stemming with NLUen.stem, de.stemOpen In Colab-
Stopwords removal with NLUstopwordsOpen In ColabStopwords
Tokenization with NLUtokenizeOpen In Colab-
Normalization of Documentsnorm_documentOpen In Colab-
Open and Closed book question answering with Google's T5en.t5 , answer_questionOpen In ColabT5-Paper, T5-Model
Overview of every task available with T5en.t5.baseOpen In ColabT5-Paper, T5-Model
Translate between more than 200 Languages in 1 line of code with Marian Modelstr.translate_to.fr, en.translate_to.fr ,fr.translate_to.he , en.translate_to.deOpen In ColabMarian-Papers, Translation-Pipeline (En to Fr), Translation-Pipeline (En to Ger)
BERT Sentence Embeddings with NLUembed_sentence.bert, pos sentiment embed_sentence.bertOpen In ColabBert-Paper, Bert Github, Bert-Sentence_Embedding
ELECTRA Sentence Embeddings with NLUembed_sentence.electra, pos sentiment embed_sentence.electraOpen In ColabElectra Paper, Sentence-Electra-Embedding
USE Sentence Embeddings with NLUuse, pos sentiment use emotionOpen In ColabUniversal Sentence Encoder, USE-TensorFlow, Sentence-USE-Embedding
Sentence similarity with NLU using BERT embeddingsembed_sentence.bert, use en.embed_sentence.electra embed_sentence.bertOpen In ColabBert-Paper, Bert Github, Bert-Sentence_Embedding
Part of Speech tagging with NLUposOpen In ColabPart of Speech
NER Aspect Airline ATISen.ner.aspect.airlineOpen In ColabNER Airline Model, Atis intent Dataset
NLU-NER_CONLL_2003_5class_examplenerOpen In ColabNER-Piple
Named-entity recognition with Deep Learning ONTO NOTESner.ontoOpen In ColabNER_Onto
Aspect based NER-Sentiment-Restaurantsen.ner.aspect_sentimentOpen In Colab-
Detect Named Entities (NER), Part of Speech Tags (POS) and Tokenize in Chinesezh.segment_words, zh.pos, zh.ner, zh.translate_to.enOpen In ColabTranslation-Pipeline (Zh to En)
Detect Named Entities (NER), Part of Speech Tags (POS) and Tokenize in Japaneseja.segment_words, ja.pos, ja.ner, ja.translate_to.enOpen In ColabTranslation-Pipeline (Ja to En)
Detect Named Entities (NER), Part of Speech Tags (POS) and Tokenize in Koreanko.segment_words, ko.pos, ko.ner.kmou.glove_840B_300d, ko.translate_to.enOpen In Colab-
Date Matchingmatch.datetimeOpen In Colab-
Typed Dependency Parsing with NLUdepOpen In ColabDependency Parsing
Untyped Dependency Parsing with NLUdep.untypedOpen In Colab-
E2E Classification with NLUe2eOpen In Colabe2e-Model
Language Classification with NLUlangOpen In Colab-
Cyberbullying Classification with NLUclassify.cyberbullyingOpen In ColabCyberbullying-Classifier
Sentiment Classification with NLU for TwitteremotionOpen In ColabEmotion detection
Fake News Classification with NLUen.classify.fakenewsOpen In ColabFakenews-Classifier
Intent Classification with NLUen.classify.intent.airlineOpen In ColabAirline-Intention classifier, Atis-Dataset
Question classification based on the TREC dataseten.classify.questionsOpen In ColabQuestion-Classifier
Sarcasm Classification with NLUen.classify.sarcasmOpen In ColabSarcasm-Classifier
Sentiment Classification with NLU for Twitteren.sentiment.twitterOpen In ColabSentiment_Twitter-Classifier
Sentiment Classification with NLU for Moviesen.sentiment.imdbOpen In ColabSentiment_imdb-Classifier
Spam Classification with NLUen.classify.spamOpen In ColabSpam-Classifier
Toxic text classification with NLUen.classify.toxicOpen In ColabToxic-Classifier
Unsupervised keyword extraction with NLU using the YAKE algorithmyakeOpen In Colab-
Grammatical Chunk Matching with NLUmatch.chunksOpen In Colab-
Getting n-Grams with NLUngramOpen In Colab-
Assertionen.med_ner.clinical en.assert, en.med_ner.clinical.biobert en.assert.biobert, ...Open In ColabHealthcare-NER, NER_Clinical-Classifier, Toxic-Classifier
De-Identification Model overviewmed_ner.jsl.wip.clinical en.de_identify, med_ner.jsl.wip.clinical en.de_identify.clinical, ...Open In ColabNER-Clinical
Drug Normalizationnorm_drugsOpen In Colab-
Entity Resolutionmed_ner.jsl.wip.clinical en.resolve_chunk.cpt_clinical, med_ner.jsl.wip.clinical en.resolve.icd10cm, ...Open In ColabNER-Clinical, Entity-Resolver clinical
Medical Named Entity Recognitionen.med_ner.ade.clinical, en.med_ner.ade.clinical_bert, en.med_ner.anatomy,en.med_ner.anatomy.biobert, ...Open In Colab-
Relation Extractionen.med_ner.jsl.wip.clinical.greedy en.relation, en.med_ner.jsl.wip.clinical.greedy en.relation.bodypart.problem, ...Open In Colab-
Visualization of NLP-Models with Spark-NLP and NLUner, dep.typed, med_ner.jsl.wip.clinical resolve_chunk.rxnorm.in, med_ner.jsl.wip.clinical resolve.icd10cmOpen In ColabNER-Piple, Dependency Parsing, NER-Clinical, Entity-Resolver (Chunks) clinical
NLU Covid-19 Emotion ShowcaseemotionOpen In GitHubEmotion detection
NLU Covid-19 Sentiment ShowcasesentimentOpen In GitHubSentiment classification
NLU Airline Emotion DemoemotionOpen In GitHubEmotion detection
NLU Airline Sentiment DemosentimentOpen In GitHubSentiment classification
Bengali NER Hindi Embeddings for 30 Modelsbn.ner, bn.lemma, ja.lemma, am.lemma, bh.lemma, en.ner.onto.bert.small_l2_128,..Open In ColabBengali-NER, Bengali-Lemmatizer, Japanese-Lemmatizer, Amharic-Lemmatizer
Entity Resolutionmed_ner.jsl.wip.clinical en.resolve.umls, med_ner.jsl.wip.clinical en.resolve.loinc, med_ner.jsl.wip.clinical en.resolve.loinc.biobertOpen In Colab-
NLU 20 Minutes Crashcourse - the fast Data Science routespell, sentiment, pos, ner, yake, en.t5, emotion, answer_question, en.t5.base ...Open In ColabT5-Model, Part of Speech, NER-Piple, Emotion detection , Spellchecker, Sentiment classification
Chapter 0: Intro: 1-linerssentiment, pos, ner, bert, elmo, embed_sentence.bertOpen In ColabPart of Speech, NER-Piple, Sentiment classification, Elmo-Embedding, Bert-Sentence_Embedding
Chapter 1: NLU base-features with some classifiers on testdataemotion, yake, stemOpen In ColabEmotion detection
Chapter 2: Translation between 300+ languages with Mariantr.translate_to.en, en.translate_to.fr, en.translate_to.heOpen In ColabTranslation-Pipeline (En to Fr), Translation (En to He)
Chapter 3: Answer questions and summarize Texts with T5answer_question, en.t5, en.t5.baseOpen In ColabT5-Model
Chapter 4: Overview of T5-Tasksen.t5.baseOpen In ColabT5-Model
Graph NLU 20 Minutes Crashcourse - State of the Art Text Mining for Graphsspell, sentiment, pos, ner, yake, emotion, med_ner.jsl.wip.clinical, ...Open In ColabPart of Speech, NER-Piple, Emotion detection, Spellchecker, Sentiment classification
Healthcare with NLUmed_ner.human_phenotype.gene_biobert, med_ner.ade_biobert, med_ner.anatomy, med_ner.bacterial_species,...Open In Colab-
Part 0: Intro: 1-linersspell, sentiment, pos, ner, bert, elmo, embed_sentence.bertOpen In ColabBert-Paper, Bert Github, T-SNE, T-SNE-Bert , Part of Speech, NER-Piple, Spellchecker, Sentiment classification, Elmo-Embedding , Bert-Sentence_Embedding
Part 1: NLU base-features with some classifiers on Testdatayake, stem, ner, emotionOpen In ColabNER-Piple, Emotion detection
Part 2: Translate between 200+ Languages in 1 line of code with Marian-Modelsen.translate_to.de, en.translate_to.fr, en.translate_to.heOpen In ColabTranslation-Pipeline (En to Fr), Translation-Pipeline (En to Ger), Translation (En to He)
Part 3: More Multilingual NLP-translations for Asian Languages with Marianen.translate_to.hi, en.translate_to.ru, en.translate_to.zhOpen In ColabTranslation (En to Hi), Translation (En to Ru), Translation (En to Zh)
Part 4: Unsupervise Chinese Keyword Extraction, NER and Translation from chinese newszh.translate_to.en, zh.segment_words, yake, zh.lemma, zh.nerOpen In ColabTranslation-Pipeline (Zh to En), Zh-Lemmatizer
Part 5: Multilingual sentiment classifier training for 100+ languagestrain.sentiment, xx.embed_sentence.labse train.sentimentn.a.Sentence_Embedding.Labse
Part 6: Question-answering and Text-summarization with T5-Modellanswer_question, en.t5, en.t5.baseOpen In ColabT5-Paper
Part 7: Overview of all tasks available with T5en.t5.baseOpen In ColabT5-Paper
Part 8: Overview of some of the Multilingual modes with State Of the Art accuracy (1-liner)bn.lemma, ja.lemma, am.lemma, bh.lemma, zh.segment_words, ...Open In ColabBengali-Lemmatizer, Japanese-Lemmatizer , Amharic-Lemmatizer
Overview of some Multilingual modes avaiable with State Of the Art accuracy (1-liner)bn.ner.cc_300d, ja.ner, zh.ner, th.ner.lst20.glove_840B_300D, ar.nerOpen In ColabBengali-NER
NLU 20 Minutes Crashcourse - the fast Data Science route-Open In Colab-

Need help?

Simple NLU Demos

Features in NLU Overview

  • Tokenization
  • Trainable Word Segmentation
  • Stop Words Removal
  • Token Normalizer
  • Document Normalizer
  • Stemmer
  • Lemmatizer
  • NGrams
  • Regex Matching
  • Text Matching,
  • Chunking
  • Date Matcher
  • Sentence Detector
  • Deep Sentence Detector (Deep learning)
  • Dependency parsing (Labeled/unlabeled)
  • Part-of-speech tagging
  • Sentiment Detection (ML models)
  • Spell Checker (ML and DL models)
  • Word Embeddings (GloVe and Word2Vec)
  • BERT Embeddings (TF Hub models)
  • ELMO Embeddings (TF Hub models)
  • ALBERT Embeddings (TF Hub models)
  • XLNet Embeddings
  • Universal Sentence Encoder (TF Hub models)
  • BERT Sentence Embeddings (42 TF Hub models)
  • Sentence Embeddings
  • Chunk Embeddings
  • Unsupervised keywords extraction
  • Language Detection & Identification (up to 375 languages)
  • Multi-class Sentiment analysis (Deep learning)
  • Multi-label Sentiment analysis (Deep learning)
  • Multi-class Text Classification (Deep learning)
  • Neural Machine Translation
  • Text-To-Text Transfer Transformer (Google T5)
  • Named entity recognition (Deep learning)
  • Easy TensorFlow integration
  • GPU Support
  • Full integration with Spark ML functions
  • 1000 pre-trained models in +200 languages!
  • Multi-lingual NER models: Arabic, Chinese, Danish, Dutch, English, Finnish, French, German, Hewbrew, Italian, Japanese, Korean, Norwegian, Persian, Polish, Portuguese, Russian, Spanish, Swedish, Urdu and more
  • Natural Language inference
  • Coreference resolution
  • Sentence Completion
  • Word sense disambiguation
  • Clinical entity recognition
  • Clinical Entity Linking
  • Entity normalization
  • Assertion Status Detection
  • De-identification
  • Relation Extraction
  • Clinical Entity Resolution

Citation

We have published a paper that you can cite for the NLU library:

@article{KOCAMAN2021100058,
    title = {Spark NLP: Natural language understanding at scale},
    journal = {Software Impacts},
    pages = {100058},
    year = {2021},
    issn = {2665-9638},
    doi = {https://doi.org/10.1016/j.simpa.2021.100058},
    url = {https://www.sciencedirect.com/science/article/pii/S2665963821000063},
    author = {Veysel Kocaman and David Talby},
    keywords = {Spark, Natural language processing, Deep learning, Tensorflow, Cluster},
    abstract = {Spark NLP is a Natural Language Processing (NLP) library built on top of Apache Spark ML. It provides simple, performant & accurate NLP annotations for machine learning pipelines that can scale easily in a distributed environment. Spark NLP comes with 1100+ pretrained pipelines and models in more than 192+ languages. It supports nearly all the NLP tasks and modules that can be used seamlessly in a cluster. Downloaded more than 2.7 million times and experiencing 9x growth since January 2020, Spark NLP is used by 54% of healthcare organizations as the world’s most widely used NLP library in the enterprise.}
    }
}

Keywords

FAQs


Did you know?

Socket installs a GitHub app to automatically flag issues on every pull request and report the health of your dependencies. Find out what is inside your node modules and prevent malicious activity before you update the dependencies.

Install

Related posts

SocketSocket SOC 2 Logo

Product

  • Package Alerts
  • Integrations
  • Docs
  • Pricing
  • FAQ
  • Roadmap

Stay in touch

Get open source security insights delivered straight into your inbox.


  • Terms
  • Privacy
  • Security

Made with ⚡️ by Socket Inc