text-preprocessor

Text Preprocessor

Normalizing texts before any natural language processing

Instalation

Using Yarn:

yarn add text-preprocessor

Or using NPM:

npm i --save text-preprocessor

Usage

const preprocessor = require('text-preprocessor');

const text = preprocessor('     that`s great!    \n \t  &  but don’t take too long okay?   \n bjŏȒk—Ɏó ');

text.clean()
    .toLowerCase()
    .unescape()
    .killUnicode()
    .normalizeSingleCurlyQuotes()
    .expandContractions();

console.log(text.toString());
// OUTPUT: "that is great! & but do not take too long okay? bjork-yo"

preprocessor(text) ⇒ TextPreprocessor

Constructs a TextPreprocessor instance

Param	Type
text	String

Methods

• TextPreprocessor
  • new TextPreprocessor(text)
  • .clean()
  • .unescape()
  • .toLowerCase()
  • .toString()
  • .expandContractions()
  • .killUnicode()
  • .replace(regexp, value)
  • .remove(regexp)
  • .removeTagsAndMentions()
  • .removeSpecialCharachters()
  • .removeURLs()
  • .removeParenthesesContents()
  • .removePunctuation()
  • .normalizeSingleCurlyQuotes()
  • .normalizeDoubleCurlyQuotes()
  • .defaults()
  • .chain()

new TextPreprocessor(text)

Normalizing texts before any natural language processing

Param	Type
text	string

textPreprocessor.clean()

and strips extra whitespace from all documents, leaving only at most one whitespace between any two other characters.

Kind: instance method of TextPreprocessor

textPreprocessor.unescape()

Converts the HTML entities &, <, >, ", and ' in string to their corresponding characters.

Kind: instance method of TextPreprocessor

textPreprocessor.toLowerCase()

Converts all the alphabetic characters in a string to lowercase.

Kind: instance method of TextPreprocessor

textPreprocessor.toString()

returns the result of chains so far

Kind: instance method of TextPreprocessor

textPreprocessor.expandContractions()

Replaces all occuring English contractions by their expanded equivalents, e.g. "don't" is changed to "do not".

Kind: instance method of TextPreprocessor

textPreprocessor.killUnicode()

Replaces hugely-ignorant, and widely subjective transliteration of latin, cryllic, greek unicode characters with english ascii.

Kind: instance method of TextPreprocessor

textPreprocessor.replace(regexp, value)

Replaces any occurrence of the given expression with the givven string

Kind: instance method of TextPreprocessor

Param	Type
regexp	RegExp
value	String

textPreprocessor.remove(regexp)

Removes any occurrence of the given expression

Kind: instance method of TextPreprocessor

Param	Type
regexp	RegExp

textPreprocessor.removeTagsAndMentions()

Removes #tags, @mentions from start of the text

Kind: instance method of TextPreprocessor

textPreprocessor.removeSpecialCharachters()

Removes all special charachters

Kind: instance method of TextPreprocessor

textPreprocessor.removeURLs()

Removes Urls and emails

Kind: instance method of TextPreprocessor

textPreprocessor.removeParenthesesContents()

Remove brackets and parentheses contents.

Kind: instance method of TextPreprocessor Example

`Hello, this is Mike (example)` to `Hello, this is Mike `

textPreprocessor.removePunctuation()

Removes punctuation from end of the text

Kind: instance method of TextPreprocessor

textPreprocessor.normalizeSingleCurlyQuotes()

Coerce single curly quotes. don’t to don't

Kind: instance method of TextPreprocessor

textPreprocessor.normalizeDoubleCurlyQuotes()

Coerce double curly quotes. it is «Khorzu” to it is "Khorzu"

Kind: instance method of TextPreprocessor

textPreprocessor.defaults()

clean,toLowerCase,unescape,killUnicode and normalizeSingleCurlyQuotes

Kind: instance method of TextPreprocessor

textPreprocessor.chain()

Executes chain of givven method names

Kind: instance method of TextPreprocessor

preprocessor(text) ⇒ TextPreprocessor

Normalizing texts before any natural language processing

Kind: global function

Param	Type
text	String