bbp

BBP - Better Boto Paginator

The boto3 module has pagination functionality. So if you're trying to enumerate a long list of resources, the paginator will provides an easier way to fetch chunk after chunk of the resource list, compared to raw list_ calls.

The problem with how the module exposes these pages is that you end up with a list of lists. For example, to get a list of all objects within an S3 bucket, you can do:

import boto3
client = boto3.client('s3')
paginator = client.get_paginator('list_objects_v2')
objects = [p['Contents'] for p in paginator.paginate(Bucket='my-bucket')]

This returns a list of lists of object information. Do you remember off the top of your head how to flatten a list of lists into one list? I sure don't. Yes I could have a for loop and append to a list each iteration, but that feels like more effort than should be required.

Even if you're not loading the whole resource list into a list in memory, and are instead processing within a for loop, you end up with a messy nested for loop.

for page in paginator.paginate(Bucket='my-bucket'):
    if ['Contents'] in page:
        for element in page['Contents']:
             process(element)

I find this a bit awkward. What I really want is:

for element in function(Bucket='my-bucket'):
   process(element)

Where function is smart enough to either return the next item on the page it already has in memory, or fetch the next page with a new API call and return the first item of that.

This library provides that function.

Installation

pip install bbp

Usage

Here's an example of how to use it for the Lambda ListFunctions paginator.

from wrapper import paginator
from pprint import pprint
for lam in paginator('lambda', 'list_functions', 'Functions'):
    pprint(lam) # process just one element at a time

• lambda is what you would pass to boto3.client()
• list_functions is what you would pass to client.get_paginator()
• Functions is the key within the response to list_objects_v2 which contains the list of resources for each page. This varies for each type of pagination call. You have to look up the documentation. Eventually I'll try to get this tool to lookup/remember that.

Here's another example, using the S3 ListObjectsV2 paginator. In this example we need to pass in the bucket name as an extra argument. Just specify this as a name=value pair at the end of the argument list.

for obj in paginator('s3', 'list_objects_v2', 'Contents', Bucket='mybucket'):
    pprint(obj) # process a single resource

• s3 is what you would pass to boto3.client()
• list_objects_v2 is what you would pass to client.get_paginator()
• Bucket='mybucket' and any other name=value arguments are what get passed to the paginator.

Packaging

This is my first ever package on PyPI. I used this guide to learn how to do this.