šŸš€ Big News: Socket Acquires Coana to Bring Reachability Analysis to Every Appsec Team.Learn more →
Socket
Book a DemoInstallSign in
Socket

html2rss-ai

Package Overview
Dependencies
Maintainers
1
Alerts
File Explorer

Advanced tools

Socket logo

Install Socket

Detect and block malicious and high-risk dependencies

Install

html2rss-ai

šŸš€ AI-powered web scraping with modern CSS support. Extract content from any website using GPT-4, handles CSS Grid/Flexbox layouts, Tailwind CSS, and complex selectors automatically.

0.3.2
PyPI
Maintainers
1

HTML2RSS AI

AI-powered universal article extractor that automatically detects and extracts article patterns from any website using OpenAI's GPT models.

Features

  • šŸ¤– AI-Powered Pattern Detection: Automatically analyzes webpage structure to find article links
  • šŸ’¾ Smart Caching: Saves patterns for reuse, reducing API calls and improving performance
  • 🐳 Docker Ready: Fully containerized with persistent storage
  • šŸ“Š Structured Output: Exports clean JSON with URLs, titles, and metadata
  • ⚔ Fast & Reliable: Handles large article listings efficiently
  • šŸ”„ Force Regeneration: Option to refresh patterns when websites change
  • šŸ“‹ Batch Processing: Process multiple URLs with JSON configuration files

Quick Start

  • Clone and setup:
git clone <repository-url>
cd html2rss-ai
cp .env.example .env
# Edit .env and set your OPENAI_API_KEY
  • Create batch configuration:
# Create config/batch_config.json with your URLs
{
  "urls": [
    {
      "url": "https://example.com/blog",
      "output_dir": "data/output/example",
      "force_regenerate": false
    }
  ]
}
  • Run batch processing:
# Process all URLs in batch_config.json
docker compose run --rm html2rss-ai

# Use custom configuration file
docker compose run --rm html2rss-ai /app/config/my_config.json
  • Access results:
  • Output files: ./data/output/
  • Pattern cache: ./pattern_cache/

šŸ“¦ Python Package

  • Install:
pip install html2rss-ai
  • Create a JSON configuration file (config.json):
{
  "urls": [
    {
      "url": "https://example.com/blog",
      "output_dir": "output",
      "force_regenerate": false
    }
  ]
}
  • Run batch processing:
export OPENAI_API_KEY="your-api-key"
python -m html2rss_ai.batch_processor config.json

Usage Examples

Basic Batch Processing

# Create configuration with Paul Graham's essays
echo '{
  "urls": [
    {
      "url": "https://www.paulgraham.com/articles.html",
      "output_dir": "data/output/paulgraham",
      "force_regenerate": false
    }
  ]
}' > config/paulgraham.json

# Process with Docker
docker compose run --rm html2rss-ai /app/config/paulgraham.json

Batch Processing with JSON Configuration

Option 1: JSON-based Batch Processing (Recommended for multiple URLs)

  • Create a configuration file (config/batch_config.json):
{
  "urls": [
    {
      "url": "https://www.paulgraham.com/articles.html",
      "output_dir": "data/output",
      "force_regenerate": false
    },
    {
      "url": "https://news.ycombinator.com",
      "output_dir": "data/output/hn",
      "force_regenerate": true
    }
  ]
}
  • Run batch processing:
# Build and run the batch processor
docker compose build html2rss-ai
docker compose run --rm html2rss-ai

# With custom configuration
docker compose run --rm html2rss-ai /app/config/my_config.json

# With error handling options
docker compose run --rm html2rss-ai /app/config/batch_config.json --continue-on-error

šŸ“– Complete Batch Processing Guide - Detailed documentation with all configuration options.

Custom Configuration

All settings are configured through the JSON configuration file:

{
  "urls": [
    {
      "url": "https://example.com/blog",
      "output_dir": "data/output/custom",
      "pattern_cache_dir": "pattern_cache/custom", 
      "force_regenerate": false,
      "save_output": true
    }
  ]
}

Configuration

Batch Processing Configuration

For processing multiple URLs, create a JSON configuration file:

{
  "urls": [
    {
      "url": "https://example.com/blog",
      "output_dir": "data/output/example", 
      "pattern_cache_dir": "pattern_cache/example",
      "force_regenerate": false,
      "save_output": true
    }
  ]
}

See docs/BATCH_PROCESSING.md for complete configuration options.

Environment Variables

VariableDefaultDescription
OPENAI_API_KEY(required)Your OpenAI API key
OUTPUT_DIRdata/outputDirectory for JSON output files
PATTERN_CACHE_DIRpattern_cacheDirectory for cached patterns

Batch Processor Arguments

# See all available options
docker compose run --rm html2rss-ai --help

# Main arguments:
config_file                Path to JSON configuration file (required)
--continue-on-error        Continue processing even if some URLs fail  
--log-level LEVEL          Set logging level (DEBUG, INFO, WARNING, ERROR)

Docker Environment

The Docker setup uses:

  • Host directories: ./data/output/ and ./pattern_cache/
  • Container paths: /app/data/output/ and /app/pattern_cache/
  • User mapping: Runs as UID/GID 1000 to avoid permission issues

Output Format

{
  "links": [
    {
      "url": "https://example.com/article-1",
      "title": "Article Title",
      "selector_used": "h2 > a"
    }
  ],
  "total_found": 42,
  "pattern_used": "articles",
  "confidence": 0.95,
  "base_url": "https://example.com/blog",
  "pattern_analysis": {
    "pattern_type": "articles",
    "primary_selectors": ["h2 > a"],
    "confidence_score": 0.95
  }
}

Development

Build Docker Image

# Build with Docker Compose (creates html2rss-ai:latest)
docker compose build

# Or build directly with custom tag
docker build -t html2rss-ai:v1.0 .

Install for Development

pip install -e ".[playwright]"
playwright install chromium

Run Tests

pytest tests/

Requirements

  • OpenAI API Key: GPT-3.5/4 access for pattern analysis
  • Docker (recommended) or Python 3.8+
  • Internet connection: For webpage scraping and API calls

License

MIT License - see LICENSE file.

Support

  • šŸ› Issues: Report bugs via GitHub Issues
  • šŸ’” Features: Suggest improvements via GitHub Discussions
  • šŸ“§ Contact: [Your contact info]

Keywords

ai-extraction

FAQs

Did you know?

Socket

Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.

Install

Related posts