irrelevant-content-detection
Advanced tools
A Python package for detecting irrelevant content in text and HTML.
Irrelevant Content Detection is a Python package for detecting and cleaning irrelevant content from text and HTML. It leverages machine learning techniques such as TF-IDF and KMeans clustering to identify and remove non-relevant information from documents.
You can install the package using pip:
pip install irrelevant-content-detection
Alternatively, you can clone the repository and install it locally:
git clone https://github.com/berkbirkan/irrelevant-content-detection.git
cd irrelevant-content-detection
pip install .
The package provides several functions to detect and clean irrelevant content from text and HTML.
The calculate_relevance_scores function calculates the TF-IDF scores for a list of texts.
from irrelevant_content_detection import calculate_relevance_scores
texts = [
"Python is a programming language.",
"This text is not relevant."
]
tfidf_scores = calculate_relevance_scores(texts)
print(tfidf_scores)
The detect_irrelevant_contents function detects irrelevant content from a list of texts.
from irrelevant_content_detection import detect_irrelevant_contents
texts = [
"Python is a programming language.",
"Python is great for data science.",
"This text is not relevant.",
"Machine learning with Python is fun.",
"Unrelated text here."
]
irrelevant_texts = detect_irrelevant_contents(texts)
print(irrelevant_texts)
The clean_irrelevant_contents function removes irrelevant content from a list of texts.
from irrelevant_content_detection import clean_irrelevant_contents
texts = [
"Python is a programming language.",
"Python is great for data science.",
"This text is not relevant.",
"Machine learning with Python is fun.",
"Unrelated text here."
]
cleaned_texts = clean_irrelevant_contents(texts)
print(cleaned_texts)
The extract_text_from_html function extracts all text from an HTML string.
from irrelevant_content_detection import extract_text_from_html
html = """
<html>
<body>
<p>Python is a programming language.</p>
<p>This text is not relevant.</p>
</body>
</html>
"""
texts = extract_text_from_html(html)
print(texts)
The detect_irrelevant_html function detects irrelevant content from an HTML string.
from irrelevant_content_detection import detect_irrelevant_html
html = """
<html>
<body>
<p>Python is a programming language.</p>
<p>Python is great for data science.</p>
<p>This text is not relevant.</p>
<p>Machine learning with Python is fun.</p>
<p>Unrelated text here.</p>
</body>
</html>
"""
irrelevant_html = detect_irrelevant_html(html)
print(irrelevant_html)
The clean_irrelevant_html function removes irrelevant content from an HTML string.
from irrelevant_content_detection import clean_irrelevant_html
html = """
<html>
<body>
<p>Python is a programming language.</p>
<p>Python is great for data science.</p>
<p>This text is not relevant.</p>
<p>Machine learning with Python is fun.</p>
<p>Unrelated text here.</p>
</body>
</html>
"""
cleaned_html = clean_irrelevant_html(html)
print(cleaned_html)
To run the tests, you can use unittest which is included in the Python Standard Library:
python -m unittest discover
Or you can run the test file directly:
python test_detector.py
Contributions are welcome! Please follow these steps to contribute:
This project is licensed under the MIT License. See the LICENSE file for more details.
FAQs
A Python package for detecting irrelevant content in text and HTML.
We found that irrelevant-content-detection demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
Newly introduced telemetry in devenv 1.4 sparked a backlash over privacy concerns, leading to the removal of its AI-powered feature after strong community pushback.
Security News
TC39 met in Seattle and advanced 9 JavaScript proposals, including three to Stage 4, introducing new features and enhancements for a future ECMAScript release.
Security News
Deno 2.2 enhances Node.js compatibility, improves dependency management, adds OpenTelemetry support, and expands linting and task automation for developers.