s3-restful

important Note

S3-restful is a modified version of the Happening gem written by Jonathan Weiss (https://github.com/peritor/happening) under the Apache 2.0 license. S3-restufl introduces compatibility patches to enable the original Happening gem to communicate with other S3 API based storage system like Cumulus and Walrus. It has also been improved to implement streaming uploads (credits go to Viktors Buls Happening fork at https://github.com/krukid/happening).

This README has been adapted from the original Happening gem. Lincense was mantained with its original authors.

---

Amazon S3 Ruby library that leverages EventMachine and em-http-request.

By using EventMachine S3-restful does not block on S3 downloads/uploads thus allowing for a higher concurrency.

S3-restful was developed by Peritor for usage inside Nanite/EventMachine. Alternatives like RightAws block during the HTTP calls thus blocking the Nanite-Agent.

For now it only supports GET, PUT and DELETE operations on S3 items. The PUT operations support S3 ACLs/permissions. S3-restful will handle redirects and retries on errors by default.

S3-restful also supports other S3 compatible APIs, like Eucalyptus Walrus and Nimbus Cumulus. See instructions below for more details.

Installation

gem install S3-restful

Usage

require 'S3-restful'

EM.run do
  item = S3Restful::S3::Item.new('bucket', 'item_id')
  item.get # non-authenticated download, works only for public-read content

  item = S3Restful::S3::Item.new('bucket', 'item_id', :aws_access_key_id => 'Your-ID', :aws_secret_access_key => 'secret')
  item.get # authenticated download
  
  item.put("The new content")
  
  item.delete
end

The above examples are a bit useless, as you never get any content back. You need to specify a callback that interacts with the http response:

EM.run do
  item = S3Restful::S3::Item.new('bucket', 'item_id', :aws_access_key_id => 'Your-ID', :aws_secret_access_key => 'secret')
  item.get do |response|
    puts "the response content is: #{response.response}"
    EM.stop
  end
end

This will enqueue your download and run it in the EventMachine event loop.

You can also react to errors:

EM.run do
  on_error = Proc.new {|response| puts "An error occured: #{response.response_header.status}"; EM.stop }
  item = S3Restful::S3::Item.new('bucket', 'item_id', :aws_access_key_id => 'Your-ID', :aws_secret_access_key => 'secret')
  item.get(:on_error => on_error) do |response|
    puts "the response content is: #{response.response}" 
    EM.stop
  end
end

If you don't supply an error handler yourself, S3-restful will be default raise an Exception.

Downloading many files could look like this:

EM.run do
  count = 100
  on_error = Proc.new {|http| puts "An error occured: #{http.response_header.status}"; EM.stop if count <= 0}
  on_success = Proc.new {|http| puts "the response is: #{http.response}"; EM.stop if count <= 0}
  
  count.times do |i|
    item = S3Restful::S3::Item.new('bucket', "item_#{i}", :aws_access_key_id => 'Your-ID', :aws_secret_access_key => 'secret')
    item.get(:on_success => on_success, :on_error => on_error)
  end
end

Upload

S3-restful supports the simple S3 PUT upload:

EM.run do
  on_error = Proc.new {|http| puts "An error occured: #{http.response_header.status}"; EM.stop }
  item = S3Restful::S3::Item.new('bucket', 'item_id', :aws_access_key_id => 'Your-ID', :aws_secret_access_key => 'secret', :on_success => on_success, :on_error => on_error)
  item.put( File.read('/etc/passwd'), :on_error => on_error ) do |response|
    puts "Upload finished!"; EM.stop 
  end
end

Setting permissions looks like this:

EM.run do
  on_error = Proc.new {|http| puts "An error occured: #{http.response_header.status}"; EM.stop }
  on_success = Proc.new {|http| puts "the response is: #{http.response}"; EM.stop }
  item = S3Restful::S3::Item.new('bucket', 'item_id', :aws_access_key_id => 'Your-ID', :aws_secret_access_key => 'secret', :permissions => 'public-write')
  item.get(:on_success => on_success, :on_error => on_error)
end

Custom headers:

EM.run do
  on_error = Proc.new {|http| puts "An error occured: #{http.response_header.status}"; EM.stop }
  on_success = Proc.new {|http| puts "the response is: #{http.response}"; EM.stop }
  item = S3Restful::S3::Item.new('bucket', 'item_id', :aws_access_key_id => 'Your-ID', :aws_secret_access_key => 'secret', :permissions => 'public-write')
  item.put(:on_success => on_success, 
           :on_error => on_error, 
           :headers => {
             'Cache-Control' => "max-age=252460800", 
             'Content-Type' => 'text/html', 
             'Expires' => 'Fri, 16 Nov 2018 22:09:29 GMT', 
             'x-amz-meta-abc' => 'ABC'
            })
end

Deleting

S3-restful support the simple S3 PUT upload:

EM.run do
  on_error = Proc.new {|response| puts "An error occured: #{response.response_header.status}"; EM.stop }
  item = S3Restful::S3::Item.new('bucket', 'item_id', :aws_access_key_id => 'Your-ID', :aws_secret_access_key => 'secret')
  item.delete(:on_error => on_error) do |response|
    puts "Deleted!"
    EM.stop
  end
end

Amazon returns no content on delete, so having a success handler is usually not needed for delete operations.

Head

You can also just load the headers of an S3 item:

EM.run do
  on_error = Proc.new {|response| puts "An error occured: #{response.response_header.status}"; EM.stop }
  item = S3Restful::S3::Item.new('bucket', 'item_id', :aws_access_key_id => 'Your-ID', :aws_secret_access_key => 'secret')
  item.head(:on_error => on_error) do |response|
    puts "Headers: #{response.inspect}"
    EM.stop
  end
end

Streaming

The response data can also be streamed:

EM.run do
  item = S3Restful::S3::Item.new( bucket...
  item.get(:on_error => on_error, :on_success => on_success ).stream do |chunk|
    # .. handle the individual chunk          
  end
end

SSL Support

S3-restful will use SSL/HTTPS by default. What it cannot do by default is verify the SSL certificate. This means that traffic is encrypted but nobody can say if the SSL-endpoint is the one you except. In order to verify the SSL certificate you need to provide S3-restful with the path to a certificate CA collection in PEM format:

S3Restful::S3.ssl_options[:cert_chain_file] = '/etc/ca-bundle.crt'

You can also set this option on each item:

S3Restful::S3::Item.new('bucket', 'item_id', 
  :aws_access_key_id => 'A', 
  :aws_secret_access_key => 'B',
  :ssl => {
    :cert_chain_file => '/etc/ca-bundle.crt'
  }
  

Or even on the request:

item.get(:ssl => {:cert_chain_file => '/etc/ca-bundle.crt'})

The SSL options are directly passed to EventMachine, see the EventMachine documentation for more information on the SSL support.

Walrus

S3-restful also supports interacting with Walrus storage from the Eucalyptus cloud. The only difference is in the Item initialization. Here you should prepend the bucket name with the 'services/Walrus/' string, as this is the base path used by the Walrus API and should be signed allong with the bucket name. You should specify the host address and port where Walrus listens for incoming requests too.

 S3Restful::S3::Item.new('services/Walrus/bucket', 'item_id', :server => 'Walrus address', :port => 8773, :protocol => 'http',
    :aws_access_key_id => 'Walrus ID', :aws_secret_access_key => 'Walrus secret')

Cumulus

S3-restful also supports interacting with Cumulus storage from the Nimbus cloud. The only difference is in the Item initialization, where you should specify the host address and port where Cumulus listens for incoming requests.

 S3Restful::S3::Item.new('bucket', 'item_id', :server => 'cumulus address', :port => 8888, :protocol => 'http',
    :aws_access_key_id => 'cumulus ID', :aws_secret_access_key => 'cumulus secret')

Credits

The AWS signing and canonical request description is based on RightAws.

License

Happening is licensed under the Apache 2.0 license. See LICENSE.txt

About

Happening was written by Jonathan Weiss for Peritor.