Huge News!Announcing our $40M Series B led by Abstract Ventures.Learn More →
customisable-language-detection
Advanced tools
customisable-language-detection - npm Package Compare versions
Comparing version 0.1.0 to 0.1.1
| { | ||
| "name": "customisable-language-detection", | ||
| "version": "0.1.0", | ||
| "description": "Library that helps detecting the language of a given piece of text.", | ||
| "version": "0.1.1", | ||
| "description": "Package to detect the language of a given text (focusing on short sms type text used on tweets, facebook, WhatsApp, etc)", | ||
| "main": "src/index.js", | ||
| "repository": { | ||
| "type": "git", | ||
| "url": "git+https://github.com/danielantelo/language-detector.git" | ||
| "url": "git+https://github.com/chattylabs/language-detector" | ||
| }, | ||
@@ -53,1 +53,2 @@ "author": { | ||
| } | ||
121
README.md
@@ -1,120 +0,5 @@ | ||
| # Language Detector | ||
| # language-detector | ||
| Package to detect the language of a given text (focusing on short "sms" text used on tweets, facebook, WhatsApp, etc) | ||
| This package aids the detection of the language of a given text. | ||
| Moved to https://github.com/chattylabs/language-detector | ||
| End goal is to detect any text, no matter how short or obscure (think messages from Twitter, WhatsApp, Instagram, SMS, etc) and return an object describing the language that best matches it. | ||
| ``` | ||
| { | ||
| language: 'en', | ||
| country: 'gb' | ||
| } | ||
| ``` | ||
| This is obtained with a combination of "reducing" and "matching". Given a piece of text we can reduce it to a set of potential languages by checking for common patterns (see `src/utils/reducers.js`), additionally we can match the n-grams of sed text to a set of pre-compiled language profiles generated through "learning" (processing known samples). | ||
| ## Usage | ||
| ### With built in language detection | ||
| Usage: | ||
| ``` | ||
| const detect = require('customisable-language-detection') | ||
| const result = detect('some text to detect') | ||
| const language = result.language | ||
| ``` | ||
| ### With custom language profiles | ||
| ``` | ||
| const detect = require('customisable-language-detection') | ||
| const customLanguageProfiles = require('../path/to/data/languageProfiles.json') | ||
| const result = detect(text, { | ||
| languageProfiles: customLanguageProfiles, | ||
| reducers: customReducers | ||
| }) | ||
| const language = result.language | ||
| ``` | ||
| NOTE: the languages you provide will be the set used, you could additionally merge them with our base: | ||
| ``` | ||
| const combinedProfiles = { | ||
| ...require('customisable-language-detection').languageProfiles, | ||
| ...customLanguageProfiles | ||
| } | ||
| ``` | ||
| #### Generating your own language profiles | ||
| You will need to build a "training" script, which analysis all your sample data and generates the language profiles object. | ||
| Your sample data should be a set of txt files containing as much text as possible and similar to the text you will be detecting. Do this per locale or language. e.g. `data/samples/en.txt`, `data/samples/fr.txt`, `data/samples/cn.txt` or `data/samples/en_GB.txt` (for country indentifier locale code must use underscore _ separator) | ||
| ``` | ||
| // bin/train.js | ||
| const train = require('customisable-language-detection').train | ||
| train('./path/to/custom/samples/*.txt', './path/to/custom/export/languageProfiles.json') | ||
| ``` | ||
| then execute it via the cli `node bin/training.js` or via an npm script. | ||
| NOTE: filenames determine the language, but using filename such as en_GB will result in the response splitting this out into language and country. | ||
| ### With custom reducers | ||
| ``` | ||
| const detect = require('customisable-language-detection') | ||
| const customLanguageProfiles = require('../path/to/data/languageProfiles.json') | ||
| const customReducers = require('../path/to/your/reducers') | ||
| const result = detect(text, { | ||
| languageProfiles: customLanguageProfiles, | ||
| reducers: customReducers | ||
| }) | ||
| const language = result.language | ||
| ``` | ||
| #### Writing reducers | ||
| Reducers are a collection of objects which map a regex to an array of languages. They help reduce the amount of languages we need to run the n-gram matching on, by finding intersections of known patterns. | ||
| So for example, imagine we provide the following reducers: | ||
| ``` | ||
| # /path/to/data/languageProfiles.json | ||
| module.exports = [ | ||
| { | ||
| regex: /[ñ]+/i, | ||
| languages: ['es', 'gn', 'gl'] | ||
| }, | ||
| { | ||
| regex: /[á|é|í|ó|ú]+/i, | ||
| languages: ['fr', 'es', 'it', 'cn', 'nl', 'fo', 'is', 'pt', 'vi', 'cy', 'el', 'gl'] | ||
| } | ||
| ] | ||
| ``` | ||
| From the above, we would reduce the words "Alimentación de niño" to the languages ['es', 'gl'], and only run n-gram matching on those. If the reducer were to just return 1 language, that would be our result. | ||
| NOTE: providing your own reducers will override the base ones. If you chose not to use them, but do use your own language profiles, languages not in your profiles will not be taken into account. | ||
| You can also combine your own reducers with the base ones: | ||
| ``` | ||
| const combinedProfiles = { | ||
| ...require('customisable-language-detection').reducers, | ||
| ...customReducers | ||
| } | ||
| ``` | ||
| ## References | ||
| - https://blog.twitter.com/engineering/en_us/a/2015/evaluating-language-identification-performance.html | ||
| - https://shuyo.wordpress.com/2012/02/21/language-detection-for-twitter-with-99-1-accuracy/ | ||
| - http://cloudmark.github.io/Language-Detection/ |
New alerts
Empty package
Supply chain riskPackage does not contain any code. It may be removed, is name squatting, or the result of a faulty package publish.
Found 1 instance in 1 package
Major refactor
Supply chain riskPackage has recently undergone a major refactor. It may be unstable or indicate significant internal changes. Use caution when updating to versions that include significant changes.
Found 1 instance in 1 package
Fixed alerts
Filesystem access
Supply chain riskAccesses the file system, and could potentially read sensitive data.
Found 1 instance in 1 package
Improved metrics
- Number of low supply chain risk alerts
- decreased by-100%
0
Worsened metrics
- Total package byte prevSize
- decreased by-99.99%
1448
- Number of package files
- decreased by-97.89%
2
- Lines of code
- decreased by-100%
0
- Number of lines in readme file
- decreased by-95.04%
6
- Number of medium supply chain risk alerts
- increased byInfinity%
2
No dependency changes