aiorequestful

aiorequestful

An asynchronous HTTP and RESTful API requests framework for asyncio and Python

• Full implementation of authorisation handling for authorising with any HTTP service, including OAuth2 flows
• Automatic response payload caching and cache retrieval on a per-endpoint basis to allow fine control over how and when response data is cached
• Automatic payload response handling to transform responses before returning and caching
• Automatic handling of common HTTP error status codes to ensure guaranteed successful requests
• Formulaic approach to retries and backoff handling to ensure smooth requests on sensitive services to handle 'Too Many Requests' style errors

Contents

• Getting Started
  • Sending simple requests
  • Handling the response payload
  • Authorising with the service
  • Caching responses
  • Handling error responses
  • Managing retries and backoff time
• Currently Supported
• Motivation and Aims
• Release History
• Contributing and Reporting Issues

NOTE:
This readme provides a brief overview of the program. Read the docs for full reference documentation.

Installation

Install through pip using one of the following commands:

pip install aiorequestful

python -m pip install aiorequestful

There are optional dependencies that you may install for optional functionality. For the current list of optional dependency groups, read the docs

Getting Started

These quick guides will help you get set up and going with aiorequestful in just a few minutes. For more detailed guides, check out the documentation.

Ultimately, the core part of this whole package is the RequestHandler. This object will handle, amongst other things, these core processes:

• creating sessions
• sending requests
• processing responses as configured
• handling error responses including backoff/retry/wait time
• authorising if configured
• caching responses if configured

Each part listed above can be configured as required. Before we get to that though, let's start with a simple example.

Sending simple requests

import asyncio
from typing import Any

from yarl import URL

from aiorequestful.request.handler import RequestHandler


async def send_get_request(handler: RequestHandler, url: str | URL) -> Any:
    """Sends a simple GET request using the given ``handler`` for the given ``url``."""
    async with handler:
        payload = await handler.get(url)

    return payload


request_handler: RequestHandler = RequestHandler.create()
api_url = "https://official-joke-api.appspot.com/jokes/programming/random"
task = send_get_request(request_handler, url=api_url)
result = asyncio.run(task)

print(result)
print(type(result).__name__)

And to send many requests, we simply do the following.

async def send_get_requests(handler: RequestHandler, url: str | URL, count: int = 20) -> tuple[Any]:
    async with handler:
        payloads = await asyncio.gather(*[handler.get(url) for _ in range(count)])

    return payloads

results = asyncio.run(send_get_requests(request_handler, url=api_url, count=20))
for result in results:
    print(result)

Here, we request some data from an open API that requires no authentication to access. Notice how the data type of the object we retrieve is a string, but we can see from the print that this is meant to be JSON data.

Handling the response payload

When we know the data type we want to retrieve, we can assign a PayloadHandler to the RequestHandler to retrieve the data type we require.

from aiorequestful.response.payload import JSONPayloadHandler

payload_handler = JSONPayloadHandler()
request_handler.payload_handler = payload_handler

task = send_get_request(request_handler, url=api_url)
result = asyncio.run(task)

print(result)
print(type(result).__name__)

By doing so, we ensure that our RequestHandler only returns data in a format that we expect. The JSONPayloadHandler is set to fail if the data given to it is not valid JSON data.

NOTE: For more info on payload handling, read the docs.

Authorising with the service

Usually, most REST APIs require a user to authenticate and authorise with their services before making any requests. We can assign an Authoriser to the RequestHandler to handle authorising for us.

from aiorequestful.auth.basic import BasicAuthoriser


async def auth_and_send_get_request(handler: RequestHandler, url: str) -> Any:
    """Authorise the ``handler`` with the service before sending a GET request to the given ``url``."""
    async with handler:
        await handler.authorise()
        payload = await handler.get(url)

    return payload


authoriser = BasicAuthoriser(login="username", password="password")
request_handler.authoriser = authoriser

task = auth_and_send_get_request(request_handler, url=api_url)
result = asyncio.run(task)

print(result)

NOTE: For more info on authorising including other types of supported authorisation flows, read the docs.

Caching responses

When requesting a large amount of requests from a REST API, you will often find it is comparatively slow for it to respond.

You may add a ResponseCache to the RequestHandler to cache the initial responses from these requests. This will help speed up future requests by hitting the cache for requests first and returning any matching response from the cache first before making a HTTP request to get the data.

from aiorequestful.cache.backend import SQLiteCache

cache = SQLiteCache.connect_with_in_memory_db()
request_handler = RequestHandler.create(cache=cache)

task = send_get_request(request_handler, url=api_url)
result = asyncio.run(task)

print(result)

However, this example will not cache anything as we have not set up repositories for the endpoints we require. Check out the documentation on caching for more info on setting up cache repositories.

NOTE: We cannot dynamically assign a cache to a instance of RequestHandler. Hence, we always need to supply the ResponseCache when instantiating the RequestHandler.

NOTE: For more info on setting a successful cache and other supported cache backends, read the docs.

Handling error responses

Often, we will receive error responses that we will need to handle. We can have the RequestHandler handle these responses by assigning StatusHandler objects.

from aiorequestful.response.status import ClientErrorStatusHandler, UnauthorisedStatusHandler, RateLimitStatusHandler

response_handlers = [
    UnauthorisedStatusHandler(), RateLimitStatusHandler(), ClientErrorStatusHandler()
]
request_handler.response_handlers = response_handlers

task = send_get_request(request_handler, url=api_url)
result = asyncio.run(task)

print(result)
print(type(result).__name__)

NOTE: For more info on StatusHandler and how they handle each response type, read the docs.

Managing retries and backoff time

Another way we can ensure a successful response is to include a retry and backoff time management strategy.

The RequestHandler provides two key mechanisms for these operations:

• The wait_timer manages the time to wait after every request whether successful or not. This is object-bound i.e. any increase to this timer affects future requests.
• The retry_timer manages the time to wait after each unsuccessful and unhandled request. This is request-bound i.e. any increase to this timer only affects the current request and not future requests.

Retries and unsuccessful backoff time

As an example, if we want to simply retry the same request 3 times without any backoff time in-between each request, we can set the following.

from aiorequestful.timer import StepCountTimer

request_handler.retry_timer = StepCountTimer(initial=0, count=3, step=0)

We set the count value to 3 for 3 retries and all other values to 0 to ensure there is no wait time between these retries.

Should we wish to add some time between each retry, we can do the following.

request_handler.retry_timer = StepCountTimer(initial=0, count=3, step=0.2)

This will now add 0.2 seconds between each unsuccessful request, waiting 0.6 seconds before the final retry for example.

This timer is generated as new for each new request so any increase in time does not carry through to future requests.

Wait backoff time

We may also wish to handle wait time after all requests. This can be useful for sensitive services that often return 'Too Many Requests' errors when making a large volume of requests at once.

from aiorequestful.timer import StepCeilingTimer

request_handler.wait_timer = StepCeilingTimer(initial=0, final=1, step=0.1)

This timer will increase by 0.1 seconds each time it is increased up to a maximum of 1 second.

WARNING: The RequestHandler is not responsible for handling when this timer is increased. A StatusHandler should be used to increase this timer such as the RateLimitStatusHandler which will increase this timer every time a 'Too Many Requests' error is returned.

This timer is the same for each new request so any increase in time does carry through to future requests.

NOTE: For more info on the available Timer objects, read the docs.

Currently Supported

• Cache Backends: SQLiteCache
• Basic Authorisation: BasicAuthoriser
• OAuth2 Flows: AuthorisationCodeFlow AuthorisationCodePKCEFlow ClientCredentialsFlow

Motivation and Aims

The key aim of this package is to provide a performant and extensible framework for interacting with REST API services and other HTTP frameworks.

As a new developer, I found it incredibly confusing understanding the myriad ways one can authenticate with a REST API, which to select for my use case, how to implement it in code and so on. I then found it a great challenge learning how to get the maximum performance from my applications for HTTP requests while balancing this against issues when accessing sensitive services which often return 'Too Many Requests' type errors as I improved the performance of my applications. As such, I separated out all the code relating to HTTP requests into this package so that other developers can use what I have learned in their applications too.

This package should implement the following:

• all possible authorisation flows for these types of services
• intelligent caching per endpoint for these responses to many common and appropriate cache backends to allow for:
  • storing of responses in a
  • reduction in request-response times by retrieving responses from the cache instead of HTTP requests
  • reducing load on sensitive HTTP-based services by hitting the cache instead, thereby reducing 'Too Many Requests' type errors
• automatic handling of common HTTP error status codes to ensure guaranteed successful requests
• other quality of life additions to ensure a large volume of responses are returned in the fastest possible time e.g. backoff/retry/wait timers

In so doing, I hope to make the access of data from these services as seamless as possible and provide the foundation of this part of the process in future applications and use cases.

Release History

For change and release history, check out the documentation.

Contributing and Reporting Issues

If you have any suggestions, wish to contribute, or have any issues to report, please do let me know via the issues tab or make a new pull request with your new feature for review.

For more info on how to contribute to aiorequestful, check out the documentation.

I hope you enjoy using aiorequestful!