felig-toolkit
Advanced tools
Readme
A toolset for Amharic Language pre-processing 🔧
Felig Toolkit Web
Now with Typescript support!
It is a toolset for Amharic Language pre-processing. It includes an Amharic Stemmer, Amharic Transliterator, Amharic Stopword remover, Amharic Lexical analyzer, Amharic Corpus indexer and Term weighter.
Breaks down Amharic language corpus and returns tokens by removing any whitespace, expanding abbreviations(አ.አ -> አዲስ አበበ), removing numbers, breaking up hyphenated words, and removing punctuation (፡ ። ! ? ...).
Removes commonly occuring words that have no contribution to the semantics of the corpus. Eg: እና ፡ ስለዚህ ፡ በመሆኑም...
Changes Unicode Amharic characters to ASCII. Exmaple: ልጆች -> ልጅኦች -> ljoc. This tool implements two types of Amharic transliteration lookup tables.
SERA (System for Ethiopic Representation in ASCII) - This system maps alphabets with similar sounds separately. Eg: (ሀ፣ሐ፡ኀ)፣(ሰ፡ሠ)፡(ጸ፡ፀ)፡(ዐ፡አ). However, in practice, these alphabets are used interchangeably and use of SERA would greatly decrease recall. NOT RECOMMENDED!
Felig - Normalizes the redundant symbols into a common symbol. RECOMMENDED!
Reduces the different morphological (e.g. inflectional or derivational) variations of Amharic word forms by taking an Amharic word and returning the stem through affix-removal with longest match.
Exmaple:
ልጆች -> ልጅኦች -> ljoc -> lj -> ልጅ
Produces an index file for the stemmed words in a corpus and relates them with the files they are found in. It also stores their frequencies per file.
Calculates the weight of words from the index file using product of their length normalized Term frequency and Inverse document frequency (tf*idf).
Felig Toolkit is available as a package on NPM for use in a Node application:
# NPM
npm install felig-toolkit
# YARN
yarn add felig-toolkit
# PNPM
pnpm install felig-toolkit
note: this package uses es-modules
import felig_toolkit from 'felig-toolkit'
felig_transliterate(word,lang): takes a single word and its' language (am/en) and returns felig-transliterated string
sera_transliterate(word,lang): takes a single word and its' language (am/en) and returns SERA-transliterated string.
rmvStopwrd(corpus): takes an Amharic corpus text (sentence/paragraph/multiple-paragraphs) and removes stop wprds
lexAnalyze(corpus): takes an Amharic corpus text returns a string of tokens
stem(word): takes an Amharic word string and returns the stem as a string (async)
indexer(filesArray, outputIndexFilePath, type): takes an array of files and produces an index (.json) file. (type= "doc" | "query")
weigh_terms(indexFilePath, outputWeightedTermsPath, typeOfIndex): takes an index file and produces a file (.json) with weighted terms. (typeOfIndex= "doc" | "query")
Felig toolkit does not work in the browser (requires node.js enviroment).
Use felig-toolkit on your server.
Exmaple:
If you are using Next.js, you can use felig-toolkit in a Next server route handler (/api/felig/route.ts) and pass the results.
felig-toolkit is open to contributions, but it is recommend to create an issue or reply in a comment to let others know what you are working on first.
npm installnode index.js on the root directoryTo prepare the following tools, these academic papers were used
FAQs
Amharic Language Pre-processor toolkit
The npm package felig-toolkit receives a total of 4 weekly downloads. As such, felig-toolkit popularity was classified as not popular.
We found that felig-toolkit demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
UnitedHealth Group disclosed that the ransomware attack on Change Healthcare compromised protected health information for millions in the U.S., with estimated costs to the company expected to reach $1 billion.
Security News
GitHub is susceptible to a CDN flaw that allows attackers to host malware on any public repository.
Security News
At Node Congress, Socket CEO Feross Aboukhadijeh uncovers the darker aspects of open source, where applications that rely heavily on third-party dependencies can be exploited in supply chain attacks.