mthrottle

mThrottle

A simple and generic rate limiter to prevent too many function calls or http requests.

When making large number of HTTP requests, its essential to respect the API limits of a server.

The Throttle class ensures you do not exceed these limits.

Python version: >= 3.7

Install with PIP

pip install mthrottle

Usage

Throttle takes two arguments - a dictionary configuration and an integer maxPenaltyCount.

The max_penalty_count is the maximum number of 429 HTTP error code, allowed to be returned during a session. Exceeding this limit will result in a runtime error.

The configuration is a dictionary which defines the limits for each endpoint. Here, RPS stands for Requests Per Second and RPM stands for Requests Per Minute. Limits are set for each end point.

Configuration must have a default key which defines the default limits.

To rate limit, run th.check method before a request or function call.

from mthrottle import Throttle

config = {
    'default': {
        'rps': 3,
        'rpm': 150
    }
    'search': {
        'rps': 20
        'rpm': 500
    }
}

maxPenaltyCount = 10

th = Throttle(config, maxPenaltyCount)

for i in range(200):
    th.check()
    print(i, flush=True, end='\r' * 3)

th.check takes a key argument, matching one of the key values in config. If key is not specified, default is used.

Example using Throttle.penalize

Throttle.penalize returns a boolean and must be called when a 429 HTTP status code is returned. It returns True when maxPenaltyCount is exceeded.

from mthrottle import Throttle
import requests

config = {
    'run': {
        'rpm': 3
    }
}

maxPenaltyCount = 10

th = Throttle(config, maxPenaltyCount)

for i in range(200):
    th.check(key='run')
    r = requests.get('https://google.com')

    if r.status_code == 429:
        if th.penalize():
            raise RuntimeError('Too many API rate limit warnings.')