requests-enhancer

requests-enhancer

A drop-in enhancement layer for the popular requests library. Adds automatic retry with exponential backoff, thread-safe LRU response caching, configurable rate limiting, and pagination helpers — with zero changes to your existing requests code.

Why requests-enhancer?

Writing robust HTTP client code means handling transient failures, avoiding hammering rate-limited APIs, caching expensive responses, and navigating paginated endpoints. requests-enhancer packages all of this into a single SmartSession class that works exactly like requests.Session.

Installation

pip install requests-enhancer

Quick Start

from requests_enhancer import SmartSession

# 3 retries, 60-second response cache, max 10 requests/second
session = SmartSession(retries=3, cache_ttl=60, rate_limit=10)
resp = session.get("https://api.example.com/users")
print(resp.json())

Features

Feature	Description
Automatic retry	Configurable attempts, exponential backoff, and status codes to retry
Response caching	Thread-safe LRU cache with per-entry TTL and explicit bypass
Rate limiting	Per-instance calls-per-second ceiling using a sliding-window limiter
Context manager	with SmartSession() as s: works out of the box
Pagination helper	paginate() generator for page- and cursor-based REST APIs
URL utilities	build_url, strip_auth, safe_json, is_json

API Reference

SmartSession

SmartSession(
    retries=3,                                    # max retry attempts
    backoff=0.5,                                  # exponential backoff factor (seconds)
    status_forcelist=(429, 500, 502, 503, 504),   # HTTP status codes that trigger retry
    cache_ttl=None,                               # GET cache TTL in seconds (None = disabled)
    cache_maxsize=256,                            # max entries in the LRU cache
    rate_limit=None,                              # max requests per second (None = disabled)
    timeout=(5, 30),                              # (connect_timeout, read_timeout) in seconds
)

All standard requests.Session methods are available: .get(), .post(), .put(), .delete(), .request().

Additional keyword argument on .request():

• use_cache=True — set to False to bypass the response cache for a single request.

Examples

from requests_enhancer import SmartSession

# Retry on server errors and rate-limit responses
session = SmartSession(retries=5, backoff=1.0, status_forcelist=(429, 500, 503))
resp = session.get("https://api.example.com/data")

# Cache responses for 5 minutes
session = SmartSession(cache_ttl=300, cache_maxsize=512)
resp1 = session.get("https://api.example.com/config")   # hits network
resp2 = session.get("https://api.example.com/config")   # served from cache

# Force a fresh request regardless of cache
fresh = session.get("https://api.example.com/config", use_cache=False)

# Rate-limited scraping — at most 2 requests/second
session = SmartSession(rate_limit=2.0, retries=3)
for url in urls:
    session.get(url)

# Context manager — session closed automatically
with SmartSession(retries=3, cache_ttl=60) as s:
    users = s.get("https://api.example.com/users").json()
    posts = s.get("https://api.example.com/posts").json()

paginate(session, url, ...)

Generator that yields requests.Response objects across paginated API endpoints.

paginate(
    session,                    # SmartSession or requests.Session instance
    url,                        # endpoint URL
    params=None,                # base query parameters
    page_param="page",          # name of the page number parameter
    per_page=100,               # items per page
    per_page_param="per_page",  # name of the page-size parameter
    max_pages=50,               # safety limit on total pages fetched
)

from requests_enhancer import SmartSession, paginate

session = SmartSession(retries=3)

all_items = []
for resp in paginate(session, "https://api.example.com/items", per_page=50):
    all_items.extend(resp.json())

print(f"Fetched {len(all_items)} items")

Pagination stops automatically when an empty page is returned, a non-OK status is received, or max_pages is reached.

URL and response utilities

from requests_enhancer import build_url, strip_auth, safe_json, is_json

# Build a URL with a path and query parameters
build_url("https://api.example.com", "/v2/users", {"page": 1, "limit": 50})
# → "https://api.example.com/v2/users?limit=50&page=1"

# Strip credentials before logging
strip_auth("https://alice:[e-mail]/data")
# → "https://api.example.com/data"

# Parse JSON safely — never raises
data = safe_json(response, default={})

# Check content type
if is_json(response):
    data = response.json()

Comparison with plain requests

Feature	requests	requests-enhancer
Automatic retry	Manual HTTPAdapter setup	✅ One parameter
Response caching	❌	✅ Thread-safe LRU with TTL
Rate limiting	❌	✅ Per-instance sliding window
Pagination	❌	✅ paginate() generator
Context manager	✅	✅ Inherited
Drop-in compatible	—	✅ Same method signatures

Development

git clone https://github.com/Hexa-devy/requests-enhancer.git
cd requests-enhancer
pip install -e ".[dev]"
pytest

Contributing

See CONTRIBUTING.md.

Changelog

See CHANGELOG.md.

License

MIT © Hexa-devy