shoryuken-later

shoryuken-later

A scheduling plugin for Shoryuken that uses Dynamo DB to delay messages arbitrarily far into the future.

Features

Supports distributed architectures

An SQS message is only queued if a conditional delete of the DDB item is successful. This eliminates any potential race condition, so if more than one shoryuken-later process is polling the same schedule table then no redundant SQS messages will be queued.

NOTE: You shouldn't really need to run more than one process, but if you do it will be safe.

One or more schedule tables

Supports polling one or more DynamoDB tables for messages.

later:
  tables:
    - default_schedule
    - other_schedule

Namespaced configuration

You can use the same configuration file for both Shoryuken and Shoryuken::Later, because the new configuration options are namespaced.

# These keys are used by both Shoryuken and Shoryuken::Later
aws:
  access_key_id:      ...       # or <%= ENV['AWS_ACCESS_KEY_ID'] %>
  secret_access_key:  ...       # or <%= ENV['AWS_SECRET_ACCESS_KEY'] %>
  region:             us-east-1 # or <%= ENV['AWS_REGION'] %>
logfile: some/path/to/file.log
  
# This key is only used by Shoryuken::Later
later:
  delay: 5 * 60   # How frequently to poll the schedule table, in seconds.
  pidfile: some/path/to/file.pid
  tables:
    - table1

# These keys are only used by Shoryuken
concurrency: 3
delay: 0
queues:
  - [queue1, 1]
  - [queue2, 2]

Usage

Starting the schedule poller

Start the shoryuken-later schedule poller with a command like:

bundle exec shoryuken-later --config shoryuken.yml

Run it as a daemon inside of your Rails app with a command like:

bundle exec shoryuken-later --config shoryuken.yml --rails --daemon

Command-line options

Integration with ActiveJob

A custom ActiveJob adapter can used to support delaying messages arbitrarily far into the future.

# config/application.rb
config.active_job.queue_adapter = :shoryuken_later

When you use the :shoryuken_later queue adapter, jobs to be performed farther than 15 minutes into the future (by setting the wait or wait_until ActiveJob options), will be inserted into the default schedule table. You can set the default schedule table in an initializer.

# config/initializers/shoryuken_later.rb
Shoryuken::Later.default_table = "#{Rails.env}_myapp_later"

Integration with Shoryuken::Worker

A new method named perform_later is added to Shoryuken::Worker allowing messages to be delayed arbitrarily far into the future. If the delay is 15 minutes or less, then the message is enqueued into the specified SQS :queue as usual. Otherwise, the message is inserted into the specified DynamoDB :schedule_table.

 require 'shoryuken-later'

 class MyWorker
   include Shoryuken::Worker
  
   shoryuken_options queue: 'default', schedule_table: 'default_schedule'
 end

 # Schedules a message to be processed 30 minutes from now.
 MyWorker.perform_later(Time.now + 30 * 60, 'Foobar')

Requirements

Ruby 2.0 or greater. Ruby 1.9 is no longer supported.

Installation

Add this line to your application's Gemfile:

gem 'shoryuken-later'

Or to get the latest updates:

gem 'shoryuken-later', github: 'joekhoobyar/shoryuken-later', branch: 'master'

And then execute:

$ bundle

Or install it yourself as:

$ gem install shoryuken-later

Documentation

Learn about using Shoryuken::Later at the Shoryuken::Later Wiki.

Learn about using Shoryuken at the Shoryuken Wiki.

Credits

Pablo Cantero, creator of Shoryuken, and everybody who contributed to it. I borrowed a lot of code from Shoryuken itself as a shortcut to making this gem.

Contributing

1. Fork it ( https://github.com/joekhoobyar/shoryuken-later/fork )
2. Create your feature branch (git checkout -b my-new-feature)
3. Commit your changes (git commit -am 'Add some feature')
4. Push to the branch (git push origin my-new-feature)
5. Create a new Pull Request