sidekiq-lock

Sidekiq::Lock

Note: This is a complete piece of software, it should work across all future sidekiq & ruby versions.

Redis-based simple locking mechanism for sidekiq. Uses SET command introduced in Redis 2.6.16.

It can be handy if you push a lot of jobs into the queue(s), but you don't want to execute specific jobs at the same time - it provides a lock method that you can use in whatever way you want.

Installation

This gem requires at least:

• redis 2.6.12
• sidekiq 6

Add this line to your application's Gemfile:

gem 'sidekiq-lock'

And then execute:

$ bundle

Usage

Sidekiq-lock is a middleware/module combination, let me go through my thought process here :).

In your worker class include Sidekiq::Lock::Worker module and provide lock attribute inside sidekiq_options, for example:

class Worker
  include Sidekiq::Worker
  include Sidekiq::Lock::Worker

  # static lock that expires after one second
  sidekiq_options lock: { timeout: 1000, name: 'lock-worker' }

  def perform
    # ...
  end
end

What will happen is:

• middleware will setup a Sidekiq::Lock::RedisLock object under Thread.current[Sidekiq::Lock::THREAD_KEY] (it should work in most use cases without any problems - but it's configurable, more below) - assuming you provided lock options, otherwise it will do nothing, just execute your worker's code

• Sidekiq::Lock::Worker module provides a lock method that just simply points to that thread variable, just as a convenience

So now in your worker class you can call (whenever you need):

• lock.acquire! - will try to acquire the lock, if returns false on failure (that means some other process / thread took the lock first)
• lock.acquired? - set to true when lock is successfully acquired
• lock.release! - deletes the lock (only if it's: acquired by current thread and not already expired)

Lock options

sidekiq_options lock will accept static values or Proc that will be called on argument(s) passed to perform method.

• timeout - specified expire time, in milliseconds
• name - name of the redis key that will be used as lock name
• value - (optional) value of the lock, if not provided it's set to random hex

Dynamic lock example:

class Worker
  include Sidekiq::Worker
  include Sidekiq::Lock::Worker
  sidekiq_options lock: {
    timeout: proc { |user_id, timeout| timeout * 2 },
    name:    proc { |user_id, timeout| "lock:peruser:#{user_id}" },
    value:   proc { |user_id, timeout| "#{user_id}" }
  }

  def perform(user_id, timeout)
    # ...
    # do some work
    # only at this point I want to acquire the lock
    if lock.acquire!
      begin
        # I can do the work
        # ...
      ensure
        # You probably want to manually release lock after work is done
        # This method can be safely called even if lock wasn't acquired
        # by current worker (thread). For more references see RedisLock class
        lock.release!
      end
    else
      # reschedule, raise an error or do whatever you want
    end
  end
end

Just be sure to provide valid redis key as a lock name.

Customizing lock method name

You can change lock to something else (globally) in sidekiq server configuration:

Sidekiq.configure_server do |config|
  config.lock_method = :redis_lock
end

Customizing lock container

If you would like to change default behavior of storing lock instance in Thread.current for whatever reason you can do that as well via server configuration:

# Any thread-safe class that implements .fetch and .store methods will do
class CustomStorage
  def fetch
    # returns stored lock instance
  end
  
  def store(lock_instance)
    # store lock
  end
end

Sidekiq.configure_server do |config|
  config.lock_container = CustomStorage.new
end

Inline testing

As you know middleware is not invoked when testing jobs inline, you can require in your test/spec helper file sidekiq/lock/testing/inline to include two methods that will help you setting / clearing up lock manually:

• set_sidekiq_lock(worker_class, payload) - note: payload should be an array of worker arguments
• clear_sidekiq_lock

Contributing

• Fork it
• Create your feature branch (git checkout -b my-new-feature)
• Commit your changes (git commit -am 'Add some feature')
• Push to the branch (git push origin my-new-feature)
• Create new Pull Request