Security News
tea.xyz Spam Plagues npm and RubyGems Package Registries
Tea.xyz, a crypto project aimed at rewarding open source contributions, is once again facing backlash due to an influx of spam packages flooding public package registries.
@lanred/gibberish-detective
Advanced tools
Readme
This utility is used to identify if a string of text contains gibberish, using a relatively simple implementation of Machine Learning and Markov Chaining. The utility is trained with a large sample of legitimate sentences with which it builds a model to determine the probability that any two characters would be found adjacently.
Options can be set during or after initialization:
const gibberish = require("gibberish-detective")({useCache: false});
gibberish.set("useCache", true)
Name | Type | Default | Description |
---|---|---|---|
useCache | bool | true | Use cache during matrix querying |
model | obj | shipped model | A learning model outputted from .train() that is used to calculate letter pairing frequencies |
thresholdFn | fn(model) | avg(baseline.good.min, baseline.bad.max) | A function that outputs a number to determine the minimum value that an .assignScore() output can be before it is considered to be gibberish |
The .detect()
(.isGibberish()
is an alias) function takes a string and returns a boolean on whether or not it detects the string as gibberish.
gibberish.detect('Luke, I am your second cousin!'); // returns false
gibberish.detect('fasdfhaiufaewroawifasdaeta'); // returns true
You may override the training model it is testing against by passing it in as the second (and optional) argument.
This package is shipped with a default model that was trained against a Sherlock Holmes novel. If necessary, a new model can be generated. As an example, if someone wishes to use this with another language, it might be prudent to generate a model based off of this language.
For best results the training text used should be decently long (at least few megabytes) with diverse and legitimate words. It is not recommended that it be trained using literature from the science fiction and fantasy genres because those selections often contain unusual word formations.
To create a new learning model, you must provide it with the aforementioned large sample of text to build the learning matrix, a sample of short lines of sentences that make sense, and a sample of lines made up of gibberish. It uses the last two data sets to form a threshold between gibberish and non-gibberish text.
let gibberish = require("gibberish-detective")();
let fs = require('fs');
let sample = fs.readFileSync('./data/good.txt', 'utf-8');
let sample_good_small = fs.readFileSync('./data/good_sm.txt', 'utf-8');
let sample_bad = fs.readFileSync('./data/bad.txt', 'utf-8');
let newModel = gibberish.train(sample, sample_good_small, sample_bad);
gibberish.set("model", newModel);
It is recommended that if re-training is necessary that this process only be done once and the resultant model be saved to a file and then thereafter that file is loaded into into the module at init-time:
At train time:
let fs = require('fs');
let newModel = gibberish.train(sample, sample_good_small, sample_bad);
fs.writeFileSync("mymodel.json", JSON.stringify(newModel))
At run-time
let myLearningModel = require('mymodel.json');
let gibberish = require("gibberish-detector")({model: myLearningModel});
rrenaud originally created a projected similar to this in python, called Gibberish-Detector.
FAQs
Detects if a string appears to be gibberish, using markov-chaining
The npm package @lanred/gibberish-detective receives a total of 1 weekly downloads. As such, @lanred/gibberish-detective popularity was classified as not popular.
We found that @lanred/gibberish-detective demonstrated a not healthy version release cadence and project activity because the last version was released a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
Tea.xyz, a crypto project aimed at rewarding open source contributions, is once again facing backlash due to an influx of spam packages flooding public package registries.
Security News
As cyber threats become more autonomous, AI-powered defenses are crucial for businesses to stay ahead of attackers who can exploit software vulnerabilities at scale.
Security News
UnitedHealth Group disclosed that the ransomware attack on Change Healthcare compromised protected health information for millions in the U.S., with estimated costs to the company expected to reach $1 billion.