obstore

obstore

The simplest, highest-throughput ¹ Python interface to S3, GCS, Azure Storage, & other S3-compliant APIs, powered by Rust.

• Sync and async API with full type hinting.
• Streaming downloads with configurable chunking.
• Streaming uploads from files or async or sync iterators.
• Streaming list, with no need to paginate.
• Automatic multipart uploads for large file objects.
• Automatic credential refresh before expiration.
• File-like object API and fsspec integration.
• Easy to install with no required Python dependencies.
• Optionally return list results in Apache Arrow format, which is faster and more memory-efficient than materializing Python dicts.
• Zero-copy data exchange between Rust and Python via the buffer protocol.

For Rust developers looking to add object_store support to their own Python packages, refer to pyo3-object_store.

Installation

To install obstore using pip:

pip install obstore

Obstore is on conda-forge and can be installed using conda, mamba, or pixi. To install obstore using conda:

conda install -c conda-forge obstore

Documentation

Full documentation is available on the website.

Head to Getting Started to dig in.

Footnotes

• Benchmarking is ongoing, but preliminary results indicate roughly 9x higher throughput than fsspec and 2.8x higher throughput than aioboto3 for many concurrent, small, get requests from an async context. ↩