Simple word tokeniser that ignores punctuation. Returning an Array of words.
tokenise(text, noStopWords). Simply returns an array of terms, without punctuation.
text is the string (text document) in which the calculations are to be performed on.noStopWords defaults to true. Set to false if you want to include stop words–e.g words such as "I" and "the".var tkn = require('tkn');
var str = "you're simply a test, a mere test";
var tokenised = tkn.tokenise(str);
>> ['simply', 'test', 'mere', 'test']
FAQs
The Token Name Service NPM package
We found that tkn demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
Research
A malicious npm package disguised as a WhatsApp client is exploiting authentication flows with a remote kill switch to exfiltrate data and destroy files.
Security News
PyPI now supports digital attestations, enhancing security and trust by allowing package maintainers to verify the authenticity of Python packages.
Security News
GitHub removed 27 malicious pull requests attempting to inject harmful code across multiple open source repositories, in another round of low-effort attacks.