Security News
New CVE Forecasting Tool Predicts 47,000 Disclosures in 2025
CVEForecast.org uses machine learning to project a record-breaking surge in vulnerability disclosures in 2025.
By Sarah Gooding - Jul 07, 2025
🚀 Big News: Socket Acquires Coana to Bring Reachability Analysis to Every Appsec Team.Learn more →
irrelevant-content-detection
Advanced tools
irrelevant-content-detection
A Python package for detecting irrelevant content in text and HTML.
- Maintainers
- 1
Irrelevant Content Detection
Irrelevant Content Detection is a Python package for detecting and cleaning irrelevant content from text and HTML. It leverages machine learning techniques such as TF-IDF and KMeans clustering to identify and remove non-relevant information from documents.
Table of Contents
Installation
You can install the package using pip:
pip install irrelevant-content-detection
Alternatively, you can clone the repository and install it locally:
git clone https://github.com/berkbirkan/irrelevant-content-detection.git
cd irrelevant-content-detection
pip install .
Usage
The package provides several functions to detect and clean irrelevant content from text and HTML.
Calculate Relevance Scores
The calculate_relevance_scores function calculates the TF-IDF scores for a list of texts.
from irrelevant_content_detection import calculate_relevance_scores
texts = [
"Python is a programming language.",
"This text is not relevant."
]
tfidf_scores = calculate_relevance_scores(texts)
print(tfidf_scores)
Detect Irrelevant Content in Text
The detect_irrelevant_contents function detects irrelevant content from a list of texts.
from irrelevant_content_detection import detect_irrelevant_contents
texts = [
"Python is a programming language.",
"Python is great for data science.",
"This text is not relevant.",
"Machine learning with Python is fun.",
"Unrelated text here."
]
irrelevant_texts = detect_irrelevant_contents(texts)
print(irrelevant_texts)
Clean Irrelevant Content from Text
The clean_irrelevant_contents function removes irrelevant content from a list of texts.
from irrelevant_content_detection import clean_irrelevant_contents
texts = [
"Python is a programming language.",
"Python is great for data science.",
"This text is not relevant.",
"Machine learning with Python is fun.",
"Unrelated text here."
]
cleaned_texts = clean_irrelevant_contents(texts)
print(cleaned_texts)
Extract Text from HTML
The extract_text_from_html function extracts all text from an HTML string.
from irrelevant_content_detection import extract_text_from_html
html = """
<html>
<body>
<p>Python is a programming language.</p>
<p>This text is not relevant.</p>
</body>
</html>
"""
texts = extract_text_from_html(html)
print(texts)
Detect Irrelevant Content in HTML
The detect_irrelevant_html function detects irrelevant content from an HTML string.
from irrelevant_content_detection import detect_irrelevant_html
html = """
<html>
<body>
<p>Python is a programming language.</p>
<p>Python is great for data science.</p>
<p>This text is not relevant.</p>
<p>Machine learning with Python is fun.</p>
<p>Unrelated text here.</p>
</body>
</html>
"""
irrelevant_html = detect_irrelevant_html(html)
print(irrelevant_html)
Clean Irrelevant Content from HTML
The clean_irrelevant_html function removes irrelevant content from an HTML string.
from irrelevant_content_detection import clean_irrelevant_html
html = """
<html>
<body>
<p>Python is a programming language.</p>
<p>Python is great for data science.</p>
<p>This text is not relevant.</p>
<p>Machine learning with Python is fun.</p>
<p>Unrelated text here.</p>
</body>
</html>
"""
cleaned_html = clean_irrelevant_html(html)
print(cleaned_html)
Testing
To run the tests, you can use unittest which is included in the Python Standard Library:
python -m unittest discover
Or you can run the test file directly:
python test_detector.py
Contributing
Contributions are welcome! Please follow these steps to contribute:
- Fork the repository.
- Create a new branch with your feature or bugfix.
- Commit your changes.
- Push to your branch.
- Create a pull request.
License
This project is licensed under the MIT License. See the LICENSE file for more details.
FAQs
A Python package for detecting irrelevant content in text and HTML.
We found that irrelevant-content-detection demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Related posts
Security News
Browserslist-rs Gets Major Refactor, Cutting Binary Size by Over 1MB
Browserslist-rs now uses static data to reduce binary size by over 1MB, improving memory use and performance for Rust-based frontend tools.
By Sarah Gooding - Jul 04, 2025
Research
Security News
8 More Malicious Firefox Extensions: Exploiting Popular Game Recognition, Hijacking User Sessions, and Stealing OAuth Credentials
Eight new malicious Firefox extensions impersonate games, steal OAuth tokens, hijack sessions, and exploit browser permissions to spy on users.
By Kush Pandya - Jul 04, 2025