Storage agnostic IO interface for humans
Developed and tested on python 3.6+
These instructions will get you a copy of the project up and running on your local machine.
pip install awscli --upgrade --user
set your aws configs
sudo apt install awscli
$ aws configure
AWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE
AWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
Default region name [None]: us-west-2
Default output format [None]: json
click here for more info about configuration and installation
Provide authentication credentials to your application code by setting the environment variable GOOGLE_APPLICATION_CREDENTIALS. Replace [FILE_NAME] with the file of the JSON file that contains your service account key
GOOGLE_APPLICATION_CREDENTIALS="/home/user/Downloads/[FILE_NAME].json"
click here for more info about configuration and installation
Provide app access token by setting the environment variable DROPBOX_TOKEN. Replace [TOKEN] with the generated token of your dropbox app
DROPBOX_TOKEN='TOKEN'
click here for more info about configuration and installation
The default installation includes only AWS connection.
pip install cloudstorageio
To add Google Drive install with drive extra:
pip install "cloudstorageio[drive]"
To add Google Storage install with google extra:
pip install "cloudstorageio[google]"
To add Dropbox install with dropbox extra:
pip install "cloudstorageio[dropbox]"
To install all extras run:
pip install "cloudstorageio[all]"
from cloudstorageio import CloudInterface
google_file_path = 'gs://bucket-name/path-to-file/sample-file.txt'
s3_file_path = 's3://bucket-name/path-to-file/sample-pic.jpg'
ci = CloudInterface()
# Write text to Google cloud storage file
with ci.open(google_file_path, 'w') as f:
f.write("Lorem Ipsum is simply dummy text")
# OR
ci.save(google_file_path, "Lorem Ipsum is simply dummy text")
# Read picture from S3 storage
with ci.open(s3_file_path, 'rb') as f:
s3_output = f.read() # binary content of picture
# OR
ci.fetch(s3_file_path)
from cloudstorageio import CloudInterface
dropbox_folder_path = 'dbx://bucket-name/folder'
s3_file_path = 's3://bucket-name/path-to-file/sample-pic.jpg'
ci = CloudInterface()
ci.isfile(s3_file_path) # returns True
ci.isdir(dropbox_folder_path) # returns True
ci.remove(s3_file_path) # removes file
ci.listdir(dropbox_folder_path) # lists folder content
from cloudstorageio import CloudInterface
dropbox_file_path = 'dbx://bucket-name/path-to-file/sample-pic.jpg'
gs_file_path = 'gs://bucket-name/path-to-file/sample-file.txt'
ci = CloudInterface()
ci.copy(from_path=dropbox_file_path, to_path=gs_file_path) # copies dropbox file to gs
from cloudstorageio import CloudInterface
s3_dir = 's3://bucket-name/sample_folder'
gs_dir = 'gs://bucket-name/sample_folder'
ci = CloudInterface()
ci.copy_dir(source_dir=s3_dir, dest_dir=gs_dir) # copies s3 folder to gs
Powered by 
Cognaize
`# Changelog All major cloudstorageio changes and pypi releases are listed here
See CHANGES references
moto to test requirementscopy_dir function in S3FAQs
Unknown package
We found that cloudstorageio demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 2 open source maintainers collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
PyPI confirms no security flaws were exploited in the Ultralytics supply chain attack and highlights improvements for safer package publishing.
Security News
Research
The Socket Research Team breaks down a malicious wrapper package that uses obfuscation to harvest credentials and exfiltrate sensitive data.
Research
Security News
Attackers used a malicious npm package typosquatting a popular ESLint plugin to steal sensitive data, execute commands, and exploit developer systems.