BucketClient
Bucket Client is a ruby gem that allows programmers to interact with popular Blob Storage cloud
services. This intends to act as a layer of abstraction, much like ORM is to databases.
With this, you may easily change the blob storage provider or even defer them.
The supported cloud storage include:
- Google Cloud Platform Cloud Storage
- Amazon Web Service S3 Bucket
- Digital Ocean Spaces
- Azure Blob Storage (Microsoft)
Installation
Add this line to your application's Gemfile:
gem 'bucket_client'
And then execute:
$ bundle
Or install it yourself as:
$ gem install bucket_client
Usage
Creation of Client
To begin using BucketClient, you have to create a Client object based on the cloud service you want to use.
It is advised to use the LocalClient for development and test, as it uses the local disk to simulate how
online blob storage will work, except for set_get_cors
and set_read_policy
functions. This ensure fast
tests and development without polluting the actual bucket.
The creation of client use the generate
class-level (or rather, module-level) method of the BucketClient
module.
Local Client
Local Client uses a local path to simulate storage of blob. The path
variable is the relative path from
terminal that it uses to simulate storage. Please be careful as it may delete files within that folder.
To create a local client:
require "bucket_client"
client = BucketClient::generate type: :local, path: "./public/sample-bucket"
client
Amazon Web Service S3 Bucket
AWS Client requires 3 values, the access id
, access key
, which is the secret, and the region
.
To create a AWS S3 client:
require "bucket_client"
client = BucketClient::generate type: :aws, id: ENV["AWS_ID"], secret: ENV["AWS_SECRET"], region: ENV["AWS_REGION"]
client
Digital Ocean Spaces
Digital Ocean spaces requires 3 values, the access id
, access key
, which is the secret, and the region
.
To create a Digital Ocean client:
require "bucket_client"
client = BucketClient::generate type: :spaces, id: ENV["AWS_ID"], secret: ENV["AWS_SECRET"], region: ENV["AWS_REGION"]
client
Azure Blob Storage
Azure Blob Storage require 2 values, the account name
and the key
, which is the secret.
To create a Azure client:
require "bucket_client"
client = BucketClient::generate type: :azure, id: ENV["AZURE_ACC_NAME"], secret: ENV["AZURE_KEY"]
client
Google Cloud Platform Storage Service
GCP Cloud Storage require 2 value, the project_id
and the secret
The secret can be passed in via 2 methods,
- serialized Hash object of the secret
- path to the secret json file on disk
To create a GCP client using the Hash object (assume the JSON value is stored as environment variable):
require "json"
require "bucket_client"
secret = JSON.parse(ENV["GOOGLE_KEY"])
client = BucketClient::generate type: :gcp, id: ENV["GOOGLE_ID"], secret: secret
client
To create a GCP client using the path to json secret:
require "bucket_client"
client = BucketClient::generate type: :gcp, id: ENV["GOOGLE_ID"], secret: "path/to/secret.json"
client
Operation Result
OperationResult is the object you obtain from the normal operations. It contains details of the operation where you
can check:
Property | Description |
---|
success | whether the operation was successful. Boolean value |
code | the status code of the operation |
message | the message of the operation. Error messages can be checked here |
value | the usable value of the operation. May be url or binary |
If you rather immediately obtain the value, and raise error when it is unsuccessful, you may use the ! version of
the method.
result = bucket.create_blob binary, "path/to/bucket/file.bin"
if result.success
p result.code
p result.message
p result.value
else
p result.code
p result.message
end
begin
result = bucket.create_blob! binary, "path/to/bucket/file.bin"
p result
rescue StandardError => e
p e.message
end
Using Client object for Bucket CRUD
The client object obtain via the generate
method can be used to perform Bucket CRUD actions. It works across
all platforms.
bang methods (methods that end with !) do not return OperationResult. Instead, they raise error if they fail.
exist_bucket key:string
=> boolean
Checks whether the bucket of a certain key exist. Raises exception when the HTTP request underneath fails
exist = client.exist_bucket "sample-bucket"
exist
create_bucket key:string
=> OperationResult<Bucket>
Creates a bucket using the provided key. Fails if bucket already exist.
value
of OperationResult
if successful is the Bucket
object that has been created.
result = client.create_bucket "sample-bucket"
if result.success
bucket = result.value
bucket
else
p result.message
p result.code
end
create_bucket! key:string
=> Bucket
Creates a bucket using the provided key. Fails if bucket already exist.
Raises exception if fail, returns Bucket
object that has been created if it succeeds
result = client.create_bucket "sample-bucket"
result
delete_bucket key:string
=> OperationResult<nil>
Deletes the bucket provided in the key. Fails if bucket does not exist.
To prevent that behaviour, use delete_bucket_if_exist
to not fail even if
bucket does not exist.
Does not return anything on success. value
will always return nil
result = client.delete_bucket "sample-bucket"
result.success
result.message
result.code
delete_bucket! key:string
=> nil
Deletes the bucket provided in the key. Fails if bucket does not exist.
To prevent that behaviour, use delete_bucket_if_exist!
to not fail even if
bucket does not exist.
Raises exception if fail. Returns nil
client.delete_bucket! "sample-bucket"
delete_bucket_if_exist key:string
=> OperationResult<nil>
Deletes the bucket provided in the key. Will succeed even if bucket does not exist.
Does not return anything on success. value
will always return nil
result = client.delete_bucket_if_exist "sample-bucket"
result.success
result.message
result.code
delete_bucket_if_exist! key:string
=> nil
Deletes the bucket provided in the key. Will succeed even if bucket does not exist.
Raises exception if the operation fails
client.delete_bucket_if_exist! "sample-bucket"
put_bucket key:string
=> OperationResult<Bucket>
Creates the bucket provided in key if it does not exist.
This method will succeed and return the bucket even if the bucket exist, unlike create_bucket
value
of the OperationResult
is the Bucket
object if the operation succeeds
result = client.put_bucket "sample-bucket"
if result.success
bucket = result.value
bucket
else
p result.message
p result.code
end
put_bucket! key:string
=> Bucket
Creates the bucket provided in key if it does not exist.
This method will succeed and return the bucket even if the bucket exist, unlike create_bucket
Returns the Bucket
object
Raises exception if the operation fails
bucket = client.put_bucket! "sample-bucket"
bucket
set_read_policy key:string
, access:symbol
=> OperationResult<nil>
Sets the read policy of the bucket. This does not work for LocalBucket
as LocalBucket
does not have
concept of "access".
Only two values are accepted: :public
and :private
.
:public
allows everyone with access to the link to read the blobs within the bucket
:private
only allows people with authorization (with secret key) to read the blob within the bucket
result = client.set_read_policy "sample-bucket", :public
result.success
result.message
result.code
set_read_policy! key:string
, access:symbol
=> nil
Sets the read policy of the bucket. This does not work for LocalBucket
as LocalBucket
does not have
concept of "access".
Raises exception if the operation fails.
Only two values are accepted: :public
and :private
.
:public
allows everyone with access to the link to read the blobs within the bucket
:private
only allows people with authorization (with secret key) to read the blob within the bucket
client.set_read_policy! "sample-bucket", :public
set_get_cors key:string
, cors:array<string>
=> OperationResult<nil>
Sets the GET CORS of the bucket. This is limits the Cross Origin Resource Sharing to the domains
within the cors
array you input. To allow all origin, please use ["*"]
as cors value.
This does not work for LocalBucket
as it does not have concept of cors
.
This is one-level higher for AzureBucket
, where it modifies the whole accounts' CORS
, not just the bucket.
result = client.set_get_cors "sample-bucket", ["*"]
result.success
result.message
result.code
set_get_cors! key:string
, cors:array<string>
=> nil
Sets the GET CORS of the bucket. This is limits the Cross Origin Resource Sharing to the domains
within the cors
array you input. To allow all origin, please use ["*"]
as cors value.
This does not work for LocalBucket
as it does not have concept of cors
.
This is one-level higher for AzureBucket
, where it modifies the whole accounts' CORS
, not just the bucket.
Raises exception if the operation fails
client.set_get_cors! "sample-bucket", ["*"]
get_bucket key:string
=> Bucket
Obtains the Bucket
instance with the key.
The bucket instance can be used to perform CRUD for blobs within the bucket.
This method will raise exception if the bucket does not exist. To improve speed as you are sure that the bucket
already exist, please use the bang version, get_bucket!
, where it will not do a look up.
bucket = get_bucket "sample-bucket"
bucket
get_bucket! key:string
=> Bucket
Obtains the Bucket
instance with the key.
The bucket instance can be used to perform CRUD for blobs within the
bucket.
This method will not do a look up, so you instance's blob CRUD operation may fail if you did not verify the existence
of this bucket. This performs faster than the non-bang version as it does not spend operation to check existence of
the bucket, making the assumption that it exist.
bucket = get_bucket! "sample-bucket"
bucket
Using Client object for Blob CRUD
The client object can perform Blob CRUD if it has access to the full URI or URL of the blob.
Bang methods (methods that end with !) do not return OperationResult. Instead, they raise error if they fail.
get_blob uri:string
=> OperationResult<array<byte>>
Obtains the binary of the blob via the URI of the blob.
value
of the OperationResult
is the byte array of the binary if the operation succeeds
result = client.get_blob "https://host.com/bucket/blob.bin"
if result.success
binary = result.value
IO.binwrite "blob.bin", binary
else
p result.message
p result.code
end
get_blob! uri:string
=> <array<byte>>
Obtains the binary of the blob via the URI of the blob
Raises exception if it fails
binary = client.get_blob! "https://host.com/bucket/blob.bin"
IO.binwrite "blob.bin", binary
exist_blob uri:string
=> boolean
Checks whether the blob exist
exist = client.exist_blob "https://host.com/bucket/blob.bin"
exist
update_blob payload:array<byte>
, uri:string
=> Operation<string>
Updates a blob with new payload in byte array
value
of the OperationResult
will return URI of the blob if success
Fails if blob with the URI doesn't exist
img = IO.binread "pic.png"
uri = "https://host.com/folder/pic.png"
result = client.update_blob img, uri
result.success
result.code
result.message
result.value
update_blob! payload:array<byte>
, uri:string
=> string
Updates a blob with new payload in byte array
Fails if blob doesnt exist
Raises exception if operation fails
img = IO.binread "pic.png"
uri = "https://host.com/folder/pic.png"
result = client.update_blob! img, uri
result
put_blob payload:array<byte>
, uri:string
=> OperationResult<string>
Creates the blob with the payload if it does not exist,
updates the blob with the new payload if it exist
value
of the OperationResult
will return URI of the blob if success
img = IO.binread "pic.png"
uri = "https://host.com/folder/pic.png"
result = client.put_blob img, uri
result.success
result.code
result.message
result.value
put_blob! payload:array<byte>
, uri:string
=> string
Creates the blob with the payload if it does not exist,
updates the blob with the new payload if it exist
Raises exception if the operation fails
img = IO.binread "pic.png"
uri = "https://host.com/folder/pic.png"
result = client.put_blob! img, uri
result
delete_blob uri:string
=> OperationResult<nil>
Deletes the blob in the provided URI
Fails if the blob does not exist. Use delete_blob_if_exist if you do not want this behaviour
value
of OperationResult
is always nil
uri = "https://host.com/folder/pic.png"
result = client.delete_blob uri
result.success
result.code
result.message
result.value
delete_blob! uri:string
=> nil
Deletes the blob in the provided URI
Fails if the blob does not exist. Use delete_blob_if_exist if you
do not want this behaviour
Raises exception if the operation fails
uri = "https://host.com/folder/pic.png"
client.delete_blob! uri
delete_blob_if_exist uri:string
=> OperationResult<nil>
Deletes the blob if it exist, else does nothing
value
of OperationResult
is always nil
uri = "https://host.com/folder/pic.png"
result = client.delete_blob uri
result.success
result.code
result.message
result.value
delete_blob_if_exist! uri:string
=> nil
Deletes the blob if it exist, else does nothing
Raises exception if the operation fails
uri = "https://host.com/folder/pic.png"
client.delete_blob! uri
Using Bucket object to perform blob CRUD with blob keys
The bucket instance is able to perform CRUD operations on blobs it owns.
Bang methods (methods that end with !) do not return OperationResult. Instead, they raise error if they fail.
In this section, we assume we obtain a Bucket
instance from the Client
instance using the get_bucket
method.
bucket = client.get_bucket! "first-bucket"
bucket
get_blob key:string
=> OperationResult<array<byte>>
Get blob as byte array
value
of the OperationResult
is the blob's byte array, if the operation succeeds
result = bucket.get_blob "image.png"
result.success
result.code
result.message
result.value
get_blob! key:string
=> array<byte>
Get blob as byte array
Raises exception if the operation fails.
img = bucket.get_blob! "image.png"
IO.binwrite "image.png", img
exist_blob key:string
=> boolean
Checks if the blob with the given key exist.
exist = bucket.exist_blob "image.png"
exist
create_blob payload:byte<array>
,key:string
=> Operation<string>
Create blob with payload. Fails if blob already exist.
value
of OperationResult
will return URI of the created blob if operations succeeded
img = IO.binread "image.png"
result = bucket.create_blob img, "image.png"
result.success
result.code
result.message
result.value
create_blob! payload:array<byte>
,key:string
=> string
Create blob with payload. Fails if blob already exist.
Raises exception if operation fails
img = IO.binread "image.png"
uri = bucket.create_blob! img, "image.png"
uri
update_blob payload:array<byte>
, key:string
=> OperationResult<string>
Updates the blob with new payload. Fails if blob does not exist
value
of OperationResult
will return URI of the created blob if operations succeeded
img = IO.binread "image.png"
result = bucket.update_blob img, "image.png"
result.success
result.code
result.message
result.value
update_blob! payload:array<byte>
, key:string
=> string
Updates the blob with new payload. Fails if blob does not exist
Raises exception if the operation fails
img = IO.binread "image.png"
result = bucket.update_blob!(img, "image.png")
result
put_blob payload:array<byte>
, key:string
=> OperationResult<string>
Creates a new blob with payload if blob does not exist. Updates blob with new payload if blob exist
value
of OperationResult
will return URI of the created blob if operations succeeded
img = IO.binread "image.png"
result = bucket.put_blob(img, "image.png")
result.success
result.code
result.message
result.value
put_blob! payload:array<byte>
, key:string
=> string
Creates a new blob with payload if blob does not exist. Updates blob with new payload if blob exist
Raises exception if operation fails
img = IO.binread "image.png"
uri = bucket.put_blob! img, "image.png"
uri
delete_blob key:string
=> Operation<nil>
Deletes a blob. Fails if the blob does not exist. To prevent this behaviour, use
delete_blob_if_exist method
value
of OperationResult
will always return nil
result = bucket.delete_blob "image.png"
result.success
result.code
result.message
result.value
delete_blob! key:string
=> nil
Deletes a blob. Fails if the blob does not exist. To prevent this behaviour, use
delete_blob_if_exist method
Raises exception if the operation fails
bucket.delete_blob! "image.png"
delete_blob_if_exist key:string
=> Operation<nil>
Deletes a blob if it exist.
value
of OperationResult
will always return nil
result = bucket.delete_blob_if_exist "image.png"
result.success
result.code
result.message
result.value
delete_blob_if_exist! key:string
=> nil
Deletes a blob if it exist.
Raises exception if the operation fails
bucket.delete_blob_if_exist! "image.png"
Using Bucket object to perform blob CRUD with full URI
The bucket instance is able to perform CRUD operations on blobs it owns, via the full URI of the blob.
Bang methods (methods that end with !) do not return OperationResult. Instead, they raise error if they fail.
In this section, we assume we obtain a Bucket
instance from the Client
instance using the get_bucket!
method.
bucket = client.get_bucket! "first-bucket"
bucket
get_blob_with_uri uri:string
=> OperationResult<array<byte>>
Get blob in target URI as byte array
value
of the OperationResult
is the blob's byte array, if the operation succeeds
result = bucket.get_blob_with_uri "https://domain.com/bucket/binary.ext"
result.success
result.code
result.message
result.value
get_blob_with_uri! key:string
=> array<byte>
Get blob in target URI as byte array
Raises exception if the operation fails.
img = bucket.get_blob_with_uri! "https://domain.com/bucket/binary.ext"
IO.binwrite "image.png", img
exist_blob_uri uri:string
=> boolean
Checks if the blob with the given URI exist.
exist = bucket.exist_blob_with_uri "https://domain.com/bucket/binary.ext"
exist
update_blob_with_uri payload:array<byte>
, uri:string
=> OperationResult<string>
Updates the blob with new payload to the uri. Fails if blob does not exist
value
of OperationResult
will return URI of the created blob if operations succeeded
img = IO.binread "image.png"
result = bucket.update_blob_with_uri img, "https://domain.com/bucket/binary.ext"
result.success
result.code
result.message
result.value
update_blob_with_uri! payload:array<byte>
, uri:string
=> string
Updates the blob with new payload. Fails if blob does not exist
Raises exception if the operation fails
img = IO.binread "image.png"
result = bucket.update_blob_with_uri! img, "https://domain.com/bucket/binary.ext"
result
put_blob_with_uri payload:array<byte>
, uri:string
=> OperationResult<string>
Creates a new blob with payload if blob does not exist. Updates blob with new payload if blob exist
value
of OperationResult
will return URI of the created blob if operations succeeded
img = IO.binread "image.png"
result = bucket.put_blob_with_uri img, "https://domain.com/bucket/binary.ext"
result.success
result.code
result.message
result.value
put_blob_with_uri! payload:array<byte>
, uri:string
=> string
Creates a new blob with payload if blob does not exist. Updates blob with new payload if blob exist
Raises exception if operation fails
img = IO.binread "image.png"
uri = bucket.put_blob_with_uri! img, "https://domain.com/bucket/binary.ext"
uri
delete_blob_with_uri uri:string
=> Operation<nil>
Deletes a blob in the uri. Fails if the blob does not exist. To prevent this behaviour, use
delete_blob_if_exist_with_uri method
value
of OperationResult
will always return nil
result = bucket.delete_blob_with_uri "https://domain.com/bucket/binary.ext"
result.success
result.code
result.message
result.value
delete_blob_with_uri! uri:string
=> nil
Deletes a blob. Fails if the blob does not exist. To prevent this behaviour, use
delete_blob_if_exist_with_uri! method
Raises exception if the operation fails
bucket.delete_blob_with_uri! "https://domain.com/bucket/binary.ext"
delete_blob_if_exist_with_uri uri:string
=> Operation<nil>
Deletes a blob in the uri if it exist.
value
of OperationResult
will always return nil
result = bucket.delete_blob_if_exist_with_uri "https://domain.com/bucket/binary.ext"
result.success
result.code
result.message
result.value
delete_blob_if_exist_with_uri! uri:string
=> nil
Deletes a blob in the uri if it exist.
Raises exception if the operation fails
bucket.delete_blob_if_exist_with_uri! "https://domain.com/bucket/binary.ext"
Development
After checking out the repo, run bin/setup
to install dependencies.
Then, run bundle exec rspec
to run the unit tests, or if you are on RubyMime, the run configuration of
Unit Test is configured. You can run the test by selecting the Unit Test
configuration.
To run the integration test with the cloud, please provide your cloud credentials in a .env
file. The
credentials required are:
Env Variable | Description |
---|
AWS_ID | The AWS access key ID |
AWS_SECRET | The AWS secret |
AWS_REGION | The 3region of the AWS bucket |
DO_ID | Digital Ocean Spaces access key ID |
DO_SECRET | Digital Ocean Spaces secret |
DO_REGION | Digital Ocean Spaces region |
AZURE_ACC_ID | The Azure Blob Storage account name |
AZURE_KEY | The Azure Blob Storage secret |
GOOGLE_ID | The project ID for the Blob Storage account |
GOOGLE_KEY | The content of the secret JSON file in 1 line |
After setting up the .env file, you can now run the command bundle exec rspec ./integration
.
Alternatively, the gitlab ci has prepared a set of blob storage account online to run integration test. You can
activate the test by pushing a commit to your branch if setting up the environment it too complicated.
To install this gem onto your local machine, run bundle exec rake install
.
Contributing
Bug reports and pull requests are welcome on GitLab at
https://gitlab.com/ruby-gem/bucket_client.
This project is intended to be a safe, welcoming space for collaboration,
and contributors are expected to adhere to the
Contributor Covenant code of conduct.
License
The gem is available as open source under the terms of the MIT License.
Code of Conduct
Everyone interacting in the BucketClient project’s codebases, issue trackers,
chat rooms and mailing lists is expected to follow the
code of conduct.