markovian-nlp

markovian-nlp

Setup

Installation

With npm installed, run terminal command:

npm i markovian-nlp

• npm package

Usage

Module import

Declare method imports at the top of each JavaScript file they will be used.

ES2015

import {
  ngramsDistribution,
  sentences,
} from 'markovian-nlp';

CommonJS

const {
  ngramsDistribution,
  sentences,
} = require('markovian-nlp');

Glossary

Learn more about computational linguistics and natural language processing (NLP) on Wikipedia.

The following terms are used in the API documentation:

term	description
bigram	2-gram sequence
deterministic	repeatable, non-random
endgram	final gram in a sequence
n-gram	contiguous gram (word) sequence
startgram	first gram in a sequence
unigram	1-gram sequence

API

ngramsDistribution(document)

View the n-grams distribution of text.

Potential applications: Markov models

Example

ngramsDistribution('birds have featured in culture and art since prehistoric times');

• test with RunKit

Output

{
  and: { _end: 0, _start: 0, art: 1 },
  art: { _end: 0, _start: 0, since: 1 },
  birds: { _end: 0, _start: 1, have: 1 },
  culture: { _end: 0, _start: 0, and: 1 },
  featured: { _end: 0, _start: 0, in: 1 },
  have: { _end: 0, _start: 0, featured: 1 },
  in: { _end: 0, _start: 0, culture: 1 },
  prehistoric: { _end: 0, _start: 0, times: 1 },
  since: { _end: 0, _start: 0, prehistoric: 1 },
  times: { _end: 1, _start: 0 },
}

Each number represents the sum of occurrences.

startgram	endgram	bigrams
"birds"	"times"	all remaining keys ("have featured", "featured in", etc.)

Input

user-defined parameter	type	implements	intermediate transformations
document	String	compromise(document)	normalization, rule-based text parsing

Return value

type	description
Object	distributions of unigrams to startgrams, endgrams, and following bigrams

Signature

// pseudocode (does not run)
ngramsDistribution(document) => ({
  ...unigrams: {
    ...{ ...bigram: bigramsDistribution },
    _end: endgramsDistribution,
    _start: startgramsDistribution,
  },
});

sentences(document)(seed)

sentences({ document[, count][, seed] })

Generate text sentences from a Markov process.

Potential applications: Natural language generation

Examples

One sentence

(example document source)

const document = "That there is constant succession and flux of ideas in our minds..."
const oneSentence = sentences(document);

Nondeterministic

oneSentence();
// output: "i have observed in the chief yet we might be able by a one
//   would promote introduce a contrary habit"

oneSentence();
// output: "this is not angry chiding or so easy to them from running away
//   with our thoughts by a proper and inure them"

• test with RunKit

Deterministic

Providing a seed produces a repeatable result:

oneSentence(1);
// deterministic output: "i would promote introduce a constant succession and hindering the path
//   and application getting the train they cannot keep their roving i would sooner reconcile
//   and contemplative part of the way to direct them"

• test with RunKit

Multiple sentences

(example document source)

sentences({
  document,
  count: 3,
  seed: 1,
});

// output: [
//   'i would promote introduce a constant succession and hindering the path and application getting the train they cannot keep their roving i would sooner reconcile and contemplative part of the way to direct them',
//   'he that train they seem to be glad to be done as may be avoided of our thoughts close to our thoughts by a proper and inure them',
//   'this wandering of attention and yet for ought i know this wandering thoughts i would promote introduce a contrary habit',
// ]

• test with RunKit

Input

user-defined parameter	type	optional	default value	implements	description
document, options.document	String	false		compromise(document)	Text.
seed, options.seed	Number	true	undefined	Chance(seed)	Leave undefined (default) for nondeterministic results, or specify seed for deterministic results.
options	Object	true
options.count	Number	true	1		Number of sentences to output.

Return value

type	description
Array[Strings...]	generated sentences