
Security News
New CVE Forecasting Tool Predicts 47,000 Disclosures in 2025
CVEForecast.org uses machine learning to project a record-breaking surge in vulnerability disclosures in 2025.
A python package for text preprocessing task in natural language processing
A python package for text preprocessing task in natural language processing.
To use this text preprocessing package, first install it using pip:
.. code-block:: python
pip install text-preprocessing
Then, import the package in your python script and call appropriate functions:
.. code-block:: python
from text_preprocessing import preprocess_text
from text_preprocessing import to_lower, remove_email, remove_url, remove_punctuation, lemmatize_word
# Preprocess text using default preprocess functions in the pipeline
text_to_process = 'Helllo, I am John Doe!!! My email is john.doe@email.com. Visit our website www.johndoe.com'
preprocessed_text = preprocess_text(text_to_process)
print(preprocessed_text)
# output: hello email visit website
# Preprocess text using custom preprocess functions in the pipeline
preprocess_functions = [to_lower, remove_email, remove_url, remove_punctuation, lemmatize_word]
preprocessed_text = preprocess_text(text_to_process, preprocess_functions)
print(preprocessed_text)
# output: helllo i am john doe my email is visit our website
.. csv-table:: :header: "Feature", "Function" :widths: 50, 35
"convert to lower case", "to_lower"
"convert to upper case", "to_upper"
"keep only alphabetic and numerical characters", "keep_alpha_numeric"
"check and correct spellings", "check_spelling"
"expand contractions", "expand_contraction"
"remove URLs", "remove_url"
"remove names", "remove_name"
"remove emails", "remove_email"
"remove phone numbers", "remove_phone_number"
"remove SSNs", "remove_ssn"
"remove credit card numbers", "remove_credit_card_number"
"remove numbers", "remove_number"
"remove bullets and numbering", "remove_itemized_bullet_and_numbering"
"remove special characters", "remove_special_character"
"remove punctuations", "remove_punctuation"
"remove extra whitespace", "remove_whitespace"
"normalize unicode (e.g., café -> cafe)", "normalize_unicode"
"remove stop words", "remove_stopword"
"tokenize words", "tokenize_word"
"tokenize sentences", "tokenize_sentence"
"substitute custom words (e.g., vs -> versus)", "substitute_token"
"stem words", "stem_word"
"lemmatize words", "lemmatize_word"
"preprocess text through a sequence of preprocessing functions", "preprocess_text"
FAQs
A python package for text preprocessing task in natural language processing
We found that text-preprocessing demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
CVEForecast.org uses machine learning to project a record-breaking surge in vulnerability disclosures in 2025.
Security News
Browserslist-rs now uses static data to reduce binary size by over 1MB, improving memory use and performance for Rust-based frontend tools.
Research
Security News
Eight new malicious Firefox extensions impersonate games, steal OAuth tokens, hijack sessions, and exploit browser permissions to spy on users.