Research
Recent Trends in Malicious Packages Targeting Discord
The Socket research team breaks down a sampling of malicious packages that download and execute files, among other suspicious behaviors, targeting the popular Discord platform.
banglanlptoolkit
Readme
Created by A F M Mahfuzul Kabir
mahfuzulkabir.com
https://www.linkedin.com/in/mahfuzulkabir
Install the requirements first with:
pip install -r requirements.txt
install the package with
pip install banglanlptoolkit
This package contains several toolkits for Bangla NLP text processing and augmentation. The available tools are listed below.
Thank you very much for using my package. I handle this package all on my own, so if there's any issue with it, I might not always be available to fix it. But if you do encounter such event, feel free to let me know and I'll fix them as soon as I can.
Bangla text normalization is a known problem in language processing for normalizing Bangla text data in computer readable format. The unicode normalization normalizes all characters of a text string in the same unicode format and removes unwanted characters present. The csebuetnlp normalizer is used for models such as BanglaBERT, BanglaT5 etc.
The package uses two normalization toolkits for Bangla text processing. The unicode normalizer is used from here. The other normalizer is specifically used for BanglaT5 translation module and taken from here.
The scarcity of good punctuation generator model for Bangla language was very dominant even a few months ago. However, with development of Bangla AI models, we now have very good punctuation generation models for our language as well.
The package uses an open-source punctuation generation model from this Kaggle dataset. I currently have this model in my huggingface for ease of use without any token. You can replace with any model of your like if you want.
The package uses three kind of text augmentation techniques.
The token replacement method uses fill-mask method to replace random tokens from a sentence and then replace them. The package uses BanglishBERT Generator model by CSEBUETNLP for this task. The model can be found in here.
The back translation method translates the sentences from Bangla to English and then to Bangla again. The package uses bn-en and en-bn models of BanglaT5 by CSEBUETNLP for this task. The models can be found here: bn2en, en2bn.
The paraphrasing toolkit uses Bangla paraphrase model of BanglaT5 by CSEBUETNLP. The model can be found in here.
The package supports both online and offline augmentations. Offline augmentation can be used to generate new dataframe of augmented texts from original dataframe. This can be saved in a variable or to a file for later use. While offline augmentation can be faster for utilizing processing power (GPU parallelism), it can get a bit annoying because of saving the augmented data every once in a while. People also love to use online augmentation, meaning, augmenting the data 'on the fly' in predefined custom dataset class. This improves performance by augmentation of sentences during training or inference, with no hassle of saving the data separately.
From version 1.1.5, I'm happy to introduce online augmentation techniques in this package. This technique was inspired from the exact same technique of torchvision.transpose, meaning, you can stack several augmentation techniques with a compose class. You can also write your own custom class of augmentation or transform techniques and use them with compose.
If you use this package, please don't forget to cite the links and papers mentioned.
FAQs
Toolkits for text processing and augmentation for Bangla NLP
We found that banglanlptoolkit demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Research
The Socket research team breaks down a sampling of malicious packages that download and execute files, among other suspicious behaviors, targeting the popular Discord platform.
Security News
Socket CEO Feross Aboukhadijeh joins a16z partners to discuss how modern, sophisticated supply chain attacks require AI-driven defenses and explore the challenges and solutions in leveraging AI for threat detection early in the development life cycle.
Security News
NIST's new AI Risk Management Framework aims to enhance the security and reliability of generative AI systems and address the unique challenges of malicious AI exploits.