obstore

obstore

The simplest, highest-throughput ¹ Python interface to S3, GCS, Azure Storage, & other S3-compliant APIs, powered by Rust.

• Sync and async API with full type hinting.
• Streaming downloads with configurable chunking.
• Streaming uploads from async or sync iterators.
• Streaming list, with no need to paginate.
• Automatic multipart uploads for large file objects.
• Support for conditional put ("put if not exists"), as well as custom tags and attributes.
• Optionally return list results in Apache Arrow format, which is faster and more memory-efficient than materializing Python dicts.
• File-like object API and fsspec integration.
• Easy to install with no required Python dependencies.
• The underlying Rust library is production quality and used in large scale production systems, such as the Rust package registry crates.io.
• Zero-copy data exchange between Rust and Python via the buffer protocol.

Installation

To install obstore using pip:

pip install obstore

Obstore is on conda-forge and can be installed using conda, mamba, or pixi. To install obstore using conda:

conda install -c conda-forge obstore

Documentation

Full documentation is available on the website.

Head to Getting Started to dig in.

Footnotes

• Benchmarking is currently ongoing, but early results indicate 9x higher throughput than aioboto3 and fsspec. ↩