scriptman

Scriptman

A powerful Python automation framework for web scraping, data processing, and task orchestration.

Features

• 🚀 Selenium Automation: Cross-platform browser automation with intelligent fallback mechanisms
• 📊 ETL Operations: Extract, Transform, Load data processing with pandas integration
• 🗄️ Database Support: Multi-database support with SQLAlchemy and pyodbc
• ⏰ Task Scheduling: Advanced task scheduling and resource management
• 🔄 Retry Mechanisms: Robust retry logic with exponential backoff
• 🧹 Cleanup Utilities: Automatic cleanup of temporary files and caches
• ⚙️ Configuration Management: Flexible configuration with TOML support

Quick Start

Installation

pip install scriptman

Basic Usage

from scriptman.powers.selenium import SeleniumInstance

# Initialize Selenium with automatic downloads directory
selenium = SeleniumInstance()

# Navigate to a website
selenium.driver.get("https://example.com")

# Download a file (will be saved to your system's Downloads folder)
# ... download logic here ...

# Wait for download to complete
downloaded_file = selenium.wait_for_downloads_to_finish()
print(f"File downloaded to: {downloaded_file}")

Downloads Directory

Scriptman automatically uses your system's default Downloads directory:

• Windows: C:\Users\<username>\Downloads (with OneDrive fallback)
• macOS: /Users/<username>/Downloads
• Linux: /home/<username>/Downloads

Smart Download Handling

Scriptman uses an intelligent download mechanism that:

• Lets Chrome use its default download directory (usually your Downloads folder)
• Monitors for completed downloads in Chrome's default location
• Automatically moves files to your configured downloads directory
• Handles filename conflicts by adding counters to duplicate names

This approach prevents download issues that can occur when forcing Chrome to use a specific download directory.

Configuration

You can customize the downloads directory in your configuration:

# scriptman.toml
[scriptman]
downloads_dir = "/path/to/custom/downloads"

File Organization

• Selenium Downloads: Files downloaded through Selenium are saved to the configured directory
• Chrome/Selenium Files: Browser executables are stored in .selenium/chrome/ subdirectory
• Temporary Files: Fallback to system temp directory if Downloads is not writable

Selenium Features

Cross-Platform Support

Scriptman's Selenium implementation includes:

• Automatic Browser Management: Downloads and manages Chrome/ChromeDriver
• Fallback Mechanisms: Graceful handling of permission issues
• Headless Mode: Optimized for server environments
• Download Monitoring: Automatic detection and relocation of completed downloads

Example: Web Scraping

from scriptman.powers.selenium import SeleniumInstance

selenium = SeleniumInstance()

# Navigate and interact
selenium.driver.get("https://example.com")
selenium.interact_with_element("//button[@id='download']", mode="click")

# Wait for download and get file path
file_path = selenium.wait_for_downloads_to_finish("report.pdf")
print(f"Downloaded: {file_path}")

Download Process Flow

• Chrome downloads to its default directory (usually Downloads)
• Scriptman monitors the default directory for new files
• File detection occurs when download completes
• Automatic move to configured directory
• Filename conflict resolution if needed
• Return final path in configured directory

ETL Operations

from scriptman.powers.etl import ETL

# Load data from various sources
data = ETL.from_csv("data.csv")
data = ETL.from_json("data.json")
data = ETL.from_db(database_handler, "SELECT * FROM table")

# Transform data
transformed = data.filter(lambda x: x['status'] == 'active')
transformed = transformed.to_snake_case()

# Save results
transformed.to_csv("output.csv")
transformed.to_db(database_handler, "output_table")

Configuration

Scriptman uses TOML configuration files:

# scriptman.toml
[scriptman]
# Downloads directory (defaults to system Downloads folder)
downloads_dir = "~/Downloads"

# Selenium settings
selenium_optimizations = true
selenium_headless = true
selenium_local_mode = true

# Logging
log_level = "INFO"

# Task settings
concurrent = true
retries = 3
task_timeout = 30

Advanced Features

Task Scheduling

from scriptman.powers.scheduler import TaskScheduler

scheduler = TaskScheduler()

# Schedule a daily task
scheduler.add_daily_task(
    "daily_report",
    task_function,
    hour=9,
    minute=0
)

# Schedule a periodic task
scheduler.add_periodic_task(
    "data_sync",
    sync_function,
    interval_minutes=30
)

Database Operations

from scriptman.powers.database import DatabaseHandler

# Connect to database
db = DatabaseHandler(
    connection_string="sqlite:///data.db"
)

# Execute queries
results = db.execute_read_query("SELECT * FROM users")
db.execute_write_query("INSERT INTO logs VALUES (?)", ["log_entry"])

Cleanup Utilities

from scriptman.powers.cleanup import CleanUp

cleaner = CleanUp()

# Clean up various resources
cleaner.cleanup()  # General cleanup
cleaner.selenium_cleanup()  # Selenium downloads
cleaner.diskcache_cleanup()  # Cache files

Development

Installation for Development

git clone <repository>
cd scriptman
pip install -e ".[dev]"

Running Tests

pytest

License

[Add your license information here]

Contributing

[Add contribution guidelines here]