resilient-requester

resilient-requester

A robust Python library for making resilient HTTP requests with retries, proxies, and custom headers.

resilient_requester is a lightweight Python package designed to simplify and improve HTTP GET requests by incorporating features such as retries, proxies, customizable headers, and timeout management.

It is built to handle network errors gracefully and provide informative logging for debugging and monitoring.

Features

• Resilient HTTP Requests: Automatically retries failed requests with configurable retry limits and randomized wait times.

• Proxy Support: Easily integrate proxies for requests.

• Customizable Headers: Generate random user-agent headers or when not provided by user.

• Timeout Management: Enforce minimum timeout values to prevent overly aggressive requests.

• Informative Logging: Detailed logging for request attempts, failures, and retries.

• Type Checking: Ensures correct parameter types to prevent runtime errors.

• Lightweight and Simple: Minimal dependencies and easy-to-use interface.

Installation

You can install resilient_requester using pip: pip install resilient_requester

Quick Start

Here’s a simple example to get started with resilient_requester:

Usage

Basic GET Request

Make a simple GET request with default settings.

ffrom resilient_requester import get_request

response = get_request("https://example.com")
if response:
    print(f"Status Code: {response.status_code}")
    print(response.text)
else:
    print("Request failed after retries.")

Using Proxies

You can specify a proxy dictionary to route requests through a proxy server.

from resilient_requester import get_request

proxy = {
    "http": "http://proxy.example.com:8080",
    "https": "https://proxy.example.com:8080"
}

response = get_request(
    url="https://example.com",
    proxy=proxy
)
if response:
    print("Request successful through proxy!")

GET Request with Custom Headers and Retries

Send a request with custom headers, a specific timeout, and multiple retries.

from resilient_requester import get_request

custom_headers = {
    "User-Agent": "MyCustomAgent/1.0",
    "Accept": "application/json"
}

response = get_request(
    url="https://api.example.com/data",
    headers=custom_headers,
    timeout=10,
    retries=3
)
if response:
    print(f"Request successful: {response.status_code}")
    print(response.json())
else:
    print("Request failed after retries.")

Timeouts and Retries

You can configure the timeout and number of retries for requests. The library enforces a minimum timeout of 5 seconds and at least 1 retry.

from resilient_requester import get_request

# Make a request with custom timeout and retries
response = get_request(
    url="https://example.com",
    timeout=10,
    retries=3
)

if response:
    print("Request successful!")
else:
    print("Request failed after 3 retries.")

API Reference

Attributes:

• headers: A dictionary containing default HTTP headers, including a random user-agent.

get_request(url, proxy=None, headers=None, timeout=0, retries=0)

Makes an HTTP GET request to the specified URL with retry logic.

Parameters:

• url (str): The URL to request.
• proxy (dict, optional): A dictionary specifying proxy settings (e.g., {"http": "http://proxy:8080"}).
• headers (dict, optional): A dictionary of HTTP headers. If not provided, defaults to headers with a random user-agent.
• timeout (int, optional): The timeout for the request in seconds. Defaults to 5 if not provided or less than 5.
• retries (int, optional): The number of retries for failed requests. Defaults to 1 if not provided or less than 1.

Returns:

• requests.Response: The response object if the request is successful.
• None: If the request fails after all retries.

Example:

response = get_request(
    url="https://example.com",
    proxy={"http": "http://proxy:8080"},
    headers={"User-Agent": "CustomAgent"},
    timeout=10,
    retries=3
)

Helper Functions

• helpers.helper_functions module contains utility functions used by the Requester class. get_headers()

  Generates default HTTP headers with a random user-agent.

  Returns:

  dict: A dictionary of HTTP headers with the following keys:

  • accept: "*/*"
    • accept-language: "en-GB,en-US;q=0.9,en;q=0.8"
    • connection: "keep-alive"
    • user-agent: A random user-agent string generated by fake_useragent.

• check_params(proxy, headers, timeout, retries): Validates and normalizes input parameters.