Multilingual tokenizer that automatically tags each token with its type. 

Multilingual tokenizer that automatically tags each token with its type

prtksxna

sanjaya

r4chn4

wink-tokenizer

Error

Ignore

Warn

License

Maintenance

Miscellaneous

Quality

Supply chain risk

Vulnerability

Admins only

Organization members

Dashboard

<upgradeLink>Upgrade</upgradeLink> to access this feature

About

Blog

Socket CLI

Customers

Socket Dependency Search

Docs

Socket for GitHub

Love

Impersonation Mode Active

Update your browser to the latest version for the best experience.

Your web browser is outdated

Use another browser for the best experience.

Your web browser is unsupported

{{count, number}} commits last two months

{{count, number}} critical severity alert

{{count, number}} critical severity alerts

{{count, number}} dependency vulnerability

{{count, number}} dependency vulnerabilities

{{count, number}} development dependencies

{{count, number}} transitive dependencies

{{count, number}} version published last month

{{count, number}} versions published last month

{{count, number}} version published in last two months

{{count, number}} versions published in last two months

{{count, number}} version published last week

{{count, number}} versions published last week

{{count, number}} version published last year

{{count, number}} versions published last year

Socket undergoes annual SOC2 Type II audits. Socket customers can request a copy of the full SOC2 Type II report.

Allows organization owners and admins to access detailed records of actions taken by members of their organization, complete with timestamps.

Prevent compromised packages from infiltrating your supply chain by monitoring for changes in real-time.

Join our Discord community to get help from Socket engineers and other Socket users.

Compare millions of open source packages on socket.dev

Detect malicious code and risky behavior (e.g. network) in your dependencies using "deep package inspection"

All data is encrypted in transit and at rest using industry standard encryption algorithms.

Get a dedicated Socket account manager and access to Socket’s Customer Success team.

Get dedicated support from Socket engineers via email and Slack.

"Shift left" with the Socket Chrome extension and give intervene early, before developers choose risky or low quality dependencies.

Catch developers using risky dependencies and give them a "speed bump" to encourage good behavior

Install the Socket GitHub app to receive real-time dependency scanning and reports with every pull request.

Integrate Socket into your GitHub SCM to prevent risky dependencies from being installed.

Socket scans for vulnerabilities in your dependencies, including CVEs, security advisories, and more.

Socket supports dependencies in many languages, with most popular ecosystems coming in 2024. We've got you covered no matter what languages you use.

Enforce license policies across your organization

Receive a Microsoft Teams message to the channel of your choice anytime Socket runs a new analysis, has a new finding, or a policy has changed.

Get help migrating from Socket's free plan to Socket's paid plans.

Run Socket on-premises or in your own cloud environment.

Query for any dependency (at any version) across your entire organization

Integrate Socket into your CI/CD pipeline to prevent risky dependencies from being deployed to production.

Analyze an entire project to find supply chain risks

Tag projects to organize and filter your projects

Restrict access based on the roles of individual users within an organization.

Receive a Slack message to the channel of your choice anytime Socket runs a new analysis, has a new finding, or a policy has changed.

Socket is SOC2 Type II compliant and undergoes annual audits.

The Socket CLI allows teams to prevent risky dependencies from being installed, and to get dependency insights from the command line.

Socket supports SAML SSO for single sign-on and identity management.

Integrate Socket into your VS Code IDE to prevent risky dependencies from being installed.

Get a POST request anytime Socket runs a new analysis, has a new finding, or a policy has changed.

Socket will analyzed your SBOMs up to a maximum number of dependencies.

API that allows you to analyze a project, lookup package scores and alerts, and more.

Create API tokens to authenticate with the Socket API and CLI.

The number of developers in your team. A developer is someone who made a commit to your organization's repositories scanned by Socket in the past month.

Use Socket to protect your private repositories.

Socket is always free for open source projects.

Socket retains all reporting data for 3 months, 1 year, or a custom term of your choice.

Access the threat feed that powers Socket.

Unlimited scale, self-hosting, and priority support for large teams.

For open-source projects, individuals, and small teams.

For growing teams with enhanced scale, security, and support.

There is a package with a similar name that is downloaded much more often.

There are packages with similar names that are downloaded much more often.

This may not be the package you want!

Instance

Alert Locations

Package

The {{ecosystem}} package {{- packageName}} receives a total of <strong>{{numWeeklyDownloadCount, number}}</strong> weekly downloads. As such, {{- packageName}} popularity was classified as <strong>{{popularity}}</strong>.

Is {{- packageName}} popular?

Is {{- packageName}} safe to use?

Is {{- packageName}} well maintained?

What is {{- packageName}}?

{{packageDescription}}Version: {{versionString}} was published by {{publisherName}}. Start using Socket to analyze {{- packageName}} and its {{dependencyCount, number}} dependencies to secure your app from supply chain attacks.

Sorry, it seems this package was removed from the registry

Package was removed

This package version has been unpublished, mostly likely due to security reasons

Package version was removed

Provide a detailed description of the malware...

Socket fights vulnerabilities and provides visibility, defense-in-depth, and proactive supply chain protection for JavaScript, Python, and Go dependencies.

Socket - Secure your dependencies. Ship with confidence.

Compare versions

{{- packageName}} - npm Package Compare versions

  "description": "Multilingual tokenizer that automatically tags each token with its type",

    "pretest": "npm run lint && npm run hint && npm run docs",

    "test": "istanbul cover _mocha ./test/",

    "coveralls": "istanbul cover _mocha --report lcovonly -- -R spec && cat ./coverage/lcov.info | coveralls && rm -rf ./coverage",

    "sourcedocs": "docker -i src -o ./sourcedocs --sidebar no",

    "docs": "documentation build src/** -f html -o docs -c docs-toc.yml",

    "hint": "jshint ./src/*.js ./test/*.js ./runkit/*.js",

    "lint": "eslint ./src/*.js ./test/*.js ./runkit/*.js"

    "url": "git+https://github.com/winkjs/wink-tokenizer.git"

    "url": "https://github.com/winkjs/wink-tokenizer/issues"

  "runkitExampleFilename": "./runkit/example.js"

//     Multilingual tokenizer that automatically tags each token with its type.

//     This file is part of “wink-sentiment”.

//     “wink-sentiment” is free software: you can redistribute

//     it and/or modify it under the terms of the GNU Affero

//     General Public License as published by the Free

//     Software Foundation, version 3 of the License.

//     “wink-sentiment” is distributed in the hope that it will

//     be useful, but WITHOUT ANY WARRANTY; without even

//     the implied warranty of MERCHANTABILITY or FITNESS

//     FOR A PARTICULAR PURPOSE.  See the GNU Affero General

//     You should have received a copy of the GNU Affero

//     General Public License along with “wink-sentiment”.

//     If not, see <http://www.gnu.org/licenses/>.

// Ordinals only for Latin like 1st, 2nd or 12th or 33rd.

var rgxOrdinalL1 = /1\dth|[04-9]th|1st|2nd|3rd|[02-9]1st|[02-9]2nd|[02-9]3rd|[02-9][04-9]th|\d+\d[04-9]th|\d+\d1st|\d+\d2nd|\d+\d3rd/g;

// Apart from detecting pure integers or decimals, also detect numbers containing

// `. - / ,` so that dates, ip address, fractions and things like codes or part

// numbers are also detected as numbers only. These regex will therefore detected

// 8.8.8.8 or 12-12-1924 or 1,1,1,1.00 or 1/4 or 1/4/66/777 as numbers.

var rgxNumberL1 = /\d+\/\d+|\d(?:[\.\,\-\/]?\d)*(?:\.\d+)?/g;

var rgxNumberDV = /[\u0966-\u096F]+\/[\u0966-\u096F]+|[\u0966-\u096F](?:[\.\,\-\/]?[\u0966-\u096F])*(?:\.[\u0966-\u096F]+)?/g;

// Devanagari Hashtags; include Latin-1 as well.

var rgxHashtagDV = /\#[\u0900-\u0963\u0970-\u097F][\u0900-\u0963\u0970-\u097F\u0966-\u096F0-9]*/gi;

var rgxEmail = /[-!#$%&'*+\/=?^\w{|}~](?:\.?[-!#$%&'*+\/=?^\w`{|}~])*@[a-z0-9](?:-?\.?[a-z0-9])*(?:\.[a-z](?:-?[a-z0-9])*)+/gi;

// Bitcoin, Ruble, Indian Rupee, Other Rupee, Dollar, Pound, Yen, Euro, Wong.

var rgxCurrency = /[\₿\₽\₹\₨\$\£\¥\€\₩]/g;

// These include both the punctuations: Latin-1 & Devanagari.

var rgxPunctuation = /[\’\'\‘\’\`\“\”\"\[\]\{\}\…\,\.\!\;\?\/\-\:\u0964\u0965]/g;

// NOTE: URL will support only EN character set for now.

var rgxURL = /(?:https?:\/\/)(?:[\da-z\.-]+)\.(?:[a-z\.]{2,6})(?:[\/\w\.\-\?#=]*)*\/?/gi;

var rgxEmoji = /[\uD800-\uDBFF][\uDC00-\uDFFF]|[\u2600-\u26FF]|[\u2700-\u27BF]/g;

var rgxEmoticon = /:-?[dps\*\/\[\]\{\}]|;-?[/(/)d]|<3/gi;

var rgxTime = /(?:\d|[01]\d|2[0-3]):?(?:[0-5][0-9])?\s?(?:[ap]m|hours|hrs)\b/gi;

// Inlcude [Latin-1 Supplement Unicode Block](https://en.wikipedia.org/wiki/Latin-1_Supplement_(Unicode_block))

var rgxWordL1 = /[a-z\u00C0-\u00D6\u00D8-\u00F6\u00F8-\u00FF]+\'[a-z\u00C0-\u00D6\u00D8-\u00F6\u00F8-\u00FF]{1,2}|[a-z\u00C0-\u00D6\u00D8-\u00F6\u00F8-\u00FF]+s\'|[a-z\u00C0-\u00D6\u00D8-\u00F6\u00F8-\u00FF]+/gi;

// Define [Devanagari Unicode Block](https://unicode.org/charts/PDF/U0900.pdf)

var rgxWordDV = /[\u0900-\u094F\u0951-\u0963\u0970-\u097F]+/gi;

var rgxSymbol = /[\u0950\~\@\#\%\^\+\=\*\|<>&]/g;

// Special regex to handle not elisions at sentence level itself. Applies to English only.

// Regexes and their categories; used for tokenizing via match/split. The

// sequence is *critical* for correct tokenization.

  { regex: rgxQuotedPhrase, category: 'quoted_phrase' },

  { regex: rgxEmail, category: 'email' },

  { regex: rgxMention, category: 'mention' },

  { regex: rgxHashtagL1, category: 'hashtag' },

  { regex: rgxHashtagDV, category: 'hashtag' },

  { regex: rgxEmoji, category: 'emoji' },

  { regex: rgxEmoticon, category: 'emoticon' },

  { regex: rgxOrdinalL1, category: 'ordinal' },

  { regex: rgxNumberL1, category: 'number' },

  { regex: rgxNumberDV, category: 'number' },

  { regex: rgxCurrency, category: 'currency' },

  { regex: rgxWordL1, category: 'word' },

  { regex: rgxWordDV, category: 'word' },

  { regex: rgxPunctuation, category: 'punctuation' },

  { regex: rgxSymbol, category: 'symbol' }

// Used to generate finger print from the tokens.

  quoted_phrase: 'q', // eslint-disable-line camelcase

 * Creates an instance of **`wink-tokenizer`**.

 * @return {methods} object conatining set of API methods for tokenizing a sentence

 * and defining configuration, plugin etc.

 * var tokenizer = require( 'wink-tokenizer' );

 * // Create your instance of wink tokenizer.

  // Default configuration: most comprehensive tokenization. Make deep copy!

  // The result of last call to `tokenize()` is retained here.

   * Attempts to tokenize the input `text` using the `rgxSplit`. The tokenization

   * is carried out by combining the regex matches and splits in the right sequence.

   * The matches are the *real tokens*, whereas splits are text units that are

   * tokenized in later rounds! The real tokens (i.e. matches) are pushed as

   * @param {string} text — unit that is to be tokenized.

   * @param {object} rgxSplit — object containing the regex and it's category.

  var tokenizeTextUnit = function ( text, rgxSplit ) {

    // Regex matches go here; note each match is a token and has the same tag

    var matches = text.match( rgxSplit.regex );

    // Balance is "what needs to be tokenized".

    var balance = text.split( rgxSplit.regex );

    // The result, in form of combination of tokens & matches, is captured here.

    // Combine tokens & matches in the following pattern [ b0 m0 b1 m1 ... ]

    matches = ( matches ) ? matches : [];

    for ( i = 0, imax = balance.length; i < imax; i += 1 ) {

		{
		"name": "wink-tokenizer",
		"version": "3.1.0",
		"version": "3.2.0",
		"description": "Multilingual tokenizer that automatically tags each token with its type",
		@@ -5,0 +5,0 @@ "keywords": [

		@@ -25,2 +25,4 @@ // wink-tokenizer
		var rgxSpaces = /\s+/g;
		// Ordinals only for Latin like 1st, 2nd or 12th or 33rd.
		var rgxOrdinalL1 = /1\dth\|[04-9]th\|1st\|2nd\|3rd\|[02-9]1st\|[02-9]2nd\|[02-9]3rd\|[02-9][04-9]th\|\d+\d[04-9]th\|\d+\d1st\|\d+\d2nd\|\d+\d3rd/g;
		// Apart from detecting pure integers or decimals, also detect numbers containing
		@@ -71,2 +73,3 @@ // `. - / ,` so that dates, ip address, fractions and things like codes or part
		{ regex: rgxTime, category: 'time' },
		{ regex: rgxOrdinalL1, category: 'ordinal' },
		{ regex: rgxNumberL1, category: 'number' },
		@@ -88,2 +91,3 @@ { regex: rgxNumberDV, category: 'number' },
		number: 'n',
		ordinal: 'o',
		quoted_phrase: 'q', // eslint-disable-line camelcase
		@@ -222,3 +226,3 @@ currency: 'r',
		* @param {object} config — It defines 0 or more properties from the list of
		* 13 properties. A true value for a property ensures tokenization
		* 14 properties. A true value for a property ensures tokenization
		* for that type of text; whereas false value will mean that the tokenization of that
		@@ -241,2 +245,3 @@ * type of text will not be attempted.
		* or 1/4 and numerals containing "`, - / .`", for example 12-12-1924 (`n`)
		* @param {boolean} [config.ordinal=true] ordinals like 1st, 2nd, 3rd, 4th or 12th or 91st (`o`)
		* @param {boolean} [config.punctuation=true] common punctuation such as `?` or `,`
		@@ -254,3 +259,3 @@ * ( token becomes fingerprint )
		* var myTokenizer.defineConfig( { mention: false } );
		* // -> 12
		* // -> 13
		* // Only tokenize words as defined above.
		@@ -257,0 +262,0 @@ * var myTokenizer.defineConfig( {} );

Improved metrics