obstore

obstore

Simple, fast integration with object storage services like Amazon S3, Google Cloud Storage, Azure Blob Storage, and S3-compliant APIs like Cloudflare R2.

• Sync and async API with full type hinting.
• Streaming downloads with configurable chunking.
• Streaming uploads from async or sync iterators.
• Streaming list, with no need to paginate.
• Automatically uses multipart uploads for large file objects.
• Support for conditional put ("put if not exists"), as well as custom tags and attributes.
• Optionally return list results as Arrow, which is faster than materializing Python dicts.
• File-like object API and fsspec integration.
• Easy to install with no required Python dependencies.
• The underlying Rust library is production quality and used in large scale production systems, such as the Rust package registry crates.io.
• Zero-copy data exchange between Rust and Python via the buffer protocol.

Installation

To install obstore using pip:

pip install obstore

Obstore is on conda-forge and can be installed using conda, mamba, or pixi. To install obstore using conda:

conda install -c conda-forge obstore

Documentation

Full documentation is available on the website.

Head to Getting Started to dig in.