sidekiq_process_killer

SidekiqProcessKiller

When you have memory leaks or "bloats" in your ruby application, identifying and fixing them can at times be a nightmare. Instead, an "acceptable" mitigation is to re-spin the workers. Its a common technique that can be found in Puma Worker Killer or Unicorn Worker Killer. Though, its neater and good practice to find and fix your leaks.

SidekiqProcessKiller plugs into Sidekiq's middleware and kills a process (by sending SIGTERM) if its processing beyond the supplied RSS threshold. Since this plugs into the middleware, the check is performed after each job.

Installation

While in testing period, this is not published to a public server, yet. Until then:

Add the following to your Gemfile

  gem "sidekiq_process_killer", git: "git://github.com/shayonj/sidekiq_process_killer.git"

Usage

Configuration

memory_threshold: 250.0 # mb
silent_mode: false
statsd_klass: nil

Config name	Description
silent_mode	When set to true, no signal will be sent to running process. This is helpful if you are planning to launch this, but want to first do a dry run.
memory_threshold	When current RSS is above this threshold, the respective Sidekiq worker will be instructed for termination (via TERM signal, which sidekiq gracefully exits).
statsd_klass	This is a class object which responds to an increment. If present, the increment function will be called with a single argument of type Hash which contains, metric_name, worker_name and current_memory_usage. This class is called when attempting to terminate a process or if the process had to be forcefully be terminated.

Updating default configuration:

class CustomMetric
  ...

  def increment(params)
    StatsD.count(
      params[:metric_name],
      tags: {
        worker_name: params[:worker_name]
      }
    )
  end

  ...
end

SidekiqProcessKiller.config do |con|
  con.memory_threshold = 1024.0
  con.silent_mode = false
  con.statsd_klass = CustomMetric.new # your custom statsd class object
end

Turn on SidekiqProcessKiller

Just plugin the Middleware

Sidekiq.configure_server do |config|
  config.server_middleware do |chain|
    chain.add SidekiqProcessKiller::Middleware
  end
end

The class tries to log as much as possible, as best as possible.