fw-storage

fw-storage

Unified file storage interface tuned for simple filtering, memory-efficiency and performance to support processing large datasets in Flywheel imports and exports.

Supported storage backends:

• fs:// - Local file-system
• s3:// - Amazon S3
• gs:// - Google Cloud Storage
• az:// - Azure Blob Storage

Installation

Add as a poetry dependency to your project:

poetry add fw-storage

Usage

from fw_storage import create_storage_client

# instantiate storage with URL
fs = create_storage_client("fs:///tmp")

# set objects from bytes, filepaths or open files
fs.set("test/file1.dat", b"content")
fs.set("test/file2.dat", "/tmp/test/file1.dat")
fs.set("test/file3.dat", open("/tmp/test/file2.dat"))

# list objects, filtering with expressions
files = list(fs.ls("test", include=["size<1kB"], exclude=["path!~file3"]))
len(files) == 2

# get object info with path, size, created and modified
info = fs.stat("test/file1.dat")
info.size == 7

# read object contents
file = fs.get("test/file1.dat")
file.read() == b"content"

# remove one or more objects
fs.rm("test", recurse=True)

Configuration

Credentials for cloud storage providers are loaded using the vendor SDKs to support every standard config file location and environment variable recommended by the provider:

Storage	Config docs
s3://	https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html
gs://	https://google-auth.readthedocs.io/en/latest/reference/google.auth.html
az://	https://docs.microsoft.com/en-us/python/api/azure-identity/azure.identity.defaultazurecredential

In addition, az:// can be configured with the envvar AZ_ACCESS_KEY.

Development

Install the project using poetry and enable pre-commit:

poetry install -E all
pre-commit install

License