Security News
Introducing the Socket Python SDK
The initial version of the Socket Python SDK is now on PyPI, enabling developers to more easily interact with the Socket REST API in Python projects.
A module for node.js and the browser that takes in text and returns text that is stripped of stopwords. Has pre-defined stopword lists for 57 languages and also takes lists with custom stopwords as input.
stopword
is a module for node and the browser that allows you to strip
stopwords from an input text. In natural language processing, "Stopwords" are
words that are so frequent that they can safely be removed from a text without
altering its meaning.
Live stopword browser demo.
sw = require('stopword')
// sw.removeStopwords and sw.[language code] now available
<script src="stopword.js"></script>
<script>
// sw.removeStopwords and sw.[language code] now available
</script>
By default, stopword
will strip an array of "meaningless" English words
sw = require('stopword')
const oldString = 'a really Interesting string with some words'.split(' ')
const newString = sw.removeStopwords(oldString)
// newString is now [ 'really', 'Interesting', 'string', 'words' ]
You can also specify a language other than English:
sw = require('stopword')
const oldString = 'Trädgårdsägare är beredda att pröva vad som helst för att bli av med de hatade mördarsniglarna åäö'.split(' ')
// sw.sv contains swedish stopwords
const newString = sw.removeStopwords(oldString, sw.sv)
// newString is now [ 'Trädgårdsägare', 'beredda', 'pröva', 'helst', 'hatade', 'mördarsniglarna', 'åäö' ]
And last, but not least, it is possible to use your own, custom list of stopwords:
sw = require('stopword')
const oldString = 'you can even roll your own custom stopword list'.split(' ')
// Just add your own list/array of stopwords
const newString = sw.removeStopwords(oldString, [ 'even', 'a', 'custom', 'stopword', 'list', 'is', 'possible']
// newString is now [ 'you', 'can', 'roll', 'your', 'own']
With spread syntax you can easily combine several stopword arrays into one. Useful for situations where two langauages are used interchangeably. Or when you have certain words that are used in every document that is not in your existing stopword arrays.
sw = require('stopword')
const oldString = 'a really interesting string with some words trädgårdsägare är beredda att pröva vad som helst för att bli av med de hatade mördarsniglarna'.split(' ')
const customStopwords = ['interesting', 'really']
const newString = sw.removeStopwords(oldString, [...sw.en, ...sw.sv, ...customStopwords]
// newString is now ['string', 'words', 'trädgårdsägare', 'beredda', 'pröva', 'helst', 'hatade', 'mördarsniglarna']
Returns an Array that represents the text with the specified stopwords removed.
text
An array of wordsstopwords
An array of stopwordssw = require('stopword')
var text = sw.removeStopwords(text[, stopwords])
// text is now an array of given words minus specified stopwords
Arrays of stopwords for the following 55 languages are supplied:
af
- Afrikaansar
- Arabic, Modern Standardhy
- Armenianeu
- Basquebn
- Bengalibr
- Bretonbg
- Bulgarianca
- Catalanzh
- Chinese Simplifiedhr
- Croatiancs
- Czechda
- Danishnl
- Dutchen
- Englisheo
- Esperantoet
- Estonianfa
- Farsifi
- Finnishfr
- Frenchgl
- Galiciande
- Germanel
- Greekha
- Hausahe
- Hebrewhi
- Hindihu
- Hungarianid
- Indonesianga
- Irishit
- Italianja
- Japaneseko
- Koreanla
- Latinlv
- Latvianlgg
- Lugbara (without diacritics)lggo
- Lugbara official (with diacritics)mr
- Marathimy
- Myanmarno
- Norwegianpl
- Polishpt
- Portugueseptbr
- Portuguese (Brazilian)pa
- Punjabi Gurmukhiro
- Romanianru
- Russiansk
- Slovaksl
- Slovenianso
- Somalist
- Sothoes
- Spanishsw
- Swahilisv
- Swedishth
- Thaitl
- Tagalog (Filipino)tr
- Turkishur
- Urduvi
- Vietnameseyo
- Yorubazu
- Zulusw = require('stopword')
norwegianStopwords = sw.no
// norwegianStopwords now contains an Array of norwgian stopwords
ja
Japanese, th
Thai and zh
Chinese Simplified and some of the other languages supported have no space between words. For these languages you need to split the text into an array of words in another way than just textString.split(' ')
. You can check out TinySegmenter for Japanese and chinese-tokenizer for Chinese.
If you can't find a stopword file for your language, you can try creating one with stopword-trainer
. We're happy to help you in the process.
Most of this work is from other projects and people, and wouldn't be possible without them. Thanks to among others the stopwords-iso project and the more-stoplist project. And thanks for all your code input: @arthurdenner, @micalevisk, @fabric-io-rodrigues, @behzadmoradi, @guysaar223, @ConnorKrammer, @GreXLin85, @nanopx and @virtual!
FAQs
A module for node.js and the browser that takes in text and returns text that is stripped of stopwords. Has pre-defined stopword lists for 62 languages and also takes lists with custom stopwords as input.
The npm package stopword receives a total of 47,810 weekly downloads. As such, stopword popularity was classified as popular.
We found that stopword demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 0 open source maintainers collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
The initial version of the Socket Python SDK is now on PyPI, enabling developers to more easily interact with the Socket REST API in Python projects.
Security News
Floating dependency ranges in npm can introduce instability and security risks into your project by allowing unverified or incompatible versions to be installed automatically, leading to unpredictable behavior and potential conflicts.
Security News
A new Rust RFC proposes "Trusted Publishing" for Crates.io, introducing short-lived access tokens via OIDC to improve security and reduce risks associated with long-lived API tokens.