markovian-nlp

markovian-nlp

Setup

Installation

With npm installed, run terminal command:

npm i markovian-nlp

• npm package

Usage

Module import

Declare method import at the top of each JavaScript file it will be used.

ES2015

import ngramsDistribution from 'markovian-nlp';

CommonJS

const ngramsDistribution = require('markovian-nlp');

Glossary

Learn more about computational linguistics and natural language processing (NLP) on Wikipedia.

The following terms are used in the API documentation:

term	description
bigram	2-gram sequence
endgram	final gram in a sequence
n-gram	contiguous gram (word) sequence
startgram	first gram in a sequence
unigram	1-gram sequence

API

ngramsDistribution(document)

View the n-grams distribution of text.

Potential applications: Markov models

Example

ngramsDistribution('birds have featured in culture and art since prehistoric times');

• test with RunKit

Output

{
  and: { _end: 0, _start: 0, art: 1 },
  art: { _end: 0, _start: 0, since: 1 },
  birds: { _end: 0, _start: 1, have: 1 },
  culture: { _end: 0, _start: 0, and: 1 },
  featured: { _end: 0, _start: 0, in: 1 },
  have: { _end: 0, _start: 0, featured: 1 },
  in: { _end: 0, _start: 0, culture: 1 },
  prehistoric: { _end: 0, _start: 0, times: 1 },
  since: { _end: 0, _start: 0, prehistoric: 1 },
  times: { _end: 1, _start: 0 },
}

Each number represents the sum of occurrences.

startgram	endgram	bigrams
"birds"	"times"	all remaining keys ("have featured", "featured in", etc.)

Input

user-defined parameter	type	implements	intermediate transformations
document	String	compromise (document)	normalization, rule-based text parsing

Return value

type	description
Object	distributions of unigrams to startgrams, endgrams, and following bigrams

Signature

// pseudocode (does not run)
ngramsDistribution(document) => ({
  ...unigrams: {
    ...{ ...bigram: bigramsDistribution },
    _end: endgramsDistribution,
    _start: startgramsDistribution,
  },
});