capistrano-sidekiq

Capistrano::Sidekiq

Sidekiq integration for Capistrano - providing systemd service management and deployment coordination for Sidekiq 7+.

Example Application

For a complete working example of this gem in action, see the capistrano-example-app which demonstrates:

• Rails 8.0 deployment with Capistrano
• Sidekiq 7.0 with Redis 7+
• Systemd service management
• rbenv integration
• Complete deployment guide

Installation

Add to your Gemfile:

gem 'capistrano-sidekiq', group: :development

Then execute:

$ bundle install

Setup

Add to your Capfile:

# Capfile
require 'capistrano/sidekiq'
install_plugin Capistrano::Sidekiq  # Default sidekiq tasks
install_plugin Capistrano::Sidekiq::Systemd  # Systemd integration

Configuration Options

Basic Settings

# config/deploy.rb
set :sidekiq_roles, :worker                  # Default role for Sidekiq processes
set :sidekiq_default_hooks, true             # Enable default deployment hooks
set :sidekiq_env, fetch(:rack_env, fetch(:rails_env, fetch(:stage)))  # Environment for Sidekiq processes

# Single config file
set :sidekiq_config_files, ['sidekiq.yml']   

# Multiple config files for different Sidekiq processes
set :sidekiq_config_files, ['sidekiq.yml', 'sidekiq-high-priority.yml']

Shared Configuration with Other Gems

This gem follows the Capistrano convention of sharing common settings across multiple service gems (e.g., capistrano-puma, capistrano-sidekiq). The following settings are shared:

• :service_unit_user - Determines if services run as system or user services
• :systemctl_bin - Path to the systemctl binary
• :lingering_user - User for systemd lingering (for user services)
• :service_unit_env_files - Shared environment files
• :service_unit_env_vars - Shared environment variables

Each gem can override these with prefixed versions (e.g., :sidekiq_systemctl_user).

Advanced Configuration

# Shared systemd settings (can be used by multiple Capistrano gems like capistrano-puma)
set :service_unit_user, :user                # Run as user or system service (:system or :user)
set :systemctl_bin, '/bin/systemctl'         # Path to systemctl binary
set :lingering_user, 'deploy'                # User to enable lingering for (defaults to :user)

# Sidekiq-specific overrides (optional - defaults to shared settings above)
set :sidekiq_systemctl_user, :system         # Override service_unit_user for Sidekiq only
set :sidekiq_systemctl_bin, '/usr/bin/systemctl'  # Override systemctl_bin for Sidekiq only
set :sidekiq_service_unit_name, "custom_sidekiq_#{fetch(:stage)}"  # Custom service name
set :sidekiq_lingering_user, 'sidekiq'       # Override lingering user for Sidekiq only

# Environment configuration
set :sidekiq_service_unit_env_files, ['/etc/environment']  # Environment files
set :sidekiq_service_unit_env_vars, [                      # Environment variables
  'RAILS_ENV=production',
  'MALLOC_ARENA_MAX=2'
]

# Logging configuration
set :sidekiq_log, -> { File.join(shared_path, 'log', 'sidekiq.log') }
set :sidekiq_error_log, -> { File.join(shared_path, 'log', 'sidekiq.error.log') }

# Command customization
set :sidekiq_command, 'sidekiq'  # Use 'sidekiq' or 'sidekiqswarm' for Enterprise
set :sidekiq_command_args, -> { "-e #{fetch(:sidekiq_env)}" }

# Use login shell to load environment (for rbenv, rvm, etc.)
set :sidekiq_use_login_shell, true

# Sidekiq Enterprise - Swarm configuration
set :sidekiq_command, 'sidekiqswarm'
set :sidekiq_swarm_env_vars, [
  'SIDEKIQ_COUNT=5',           # Number of processes
  'SIDEKIQ_MAXMEM_MB=768',     # Memory limit per process
  'SIDEKIQ_PRELOAD_APP=1'      # Preload app for memory efficiency
]
# Merge swarm env vars with regular env vars
set :sidekiq_service_unit_env_vars, -> { 
  fetch(:sidekiq_service_unit_env_vars, []) + fetch(:sidekiq_swarm_env_vars, [])
}

Per-Server Configuration

You can configure Sidekiq differently for specific servers, allowing you to run different Sidekiq processes with different configurations on different servers.

Basic Per-Server Setup

# config/deploy/production.rb
server 'worker1.example.com', 
  roles: [:worker], 
  sidekiq_config_files: ['sidekiq_high_priority.yml']

server 'worker2.example.com', 
  roles: [:worker], 
  sidekiq_config_files: ['sidekiq_low_priority.yml']

Advanced Per-Server Configuration

# Different users and multiple processes per server
server 'worker1.example.com',
  roles: [:worker],
  sidekiq_config_files: ['sidekiq_critical.yml', 'sidekiq_default.yml'],
  sidekiq_user: 'sidekiq_critical',
  sidekiq_systemctl_user: :system  # Run as system service on this server

server 'worker2.example.com',
  roles: [:worker],
  sidekiq_config_files: ['sidekiq_batch.yml'],
  sidekiq_user: 'sidekiq_batch',
  sidekiq_service_unit_env_vars: ['MALLOC_ARENA_MAX=4']  # Server-specific env vars

How It Works

• Configuration Files: Each server can have its own set of sidekiq_config_files
• Service Creation: A separate systemd service is created for each config file
• Service Naming: Services are named as <app>_sidekiq_<stage> for the default sidekiq.yml, or <app>_sidekiq_<stage>.<config_name> for additional configs
• Independent Control: Each service can be started, stopped, and restarted independently

Example Configurations

config/sidekiq_high_priority.yml:

:concurrency: 10
:queues:
  - [critical, 2]
  - [high, 1]

config/sidekiq_low_priority.yml:

:concurrency: 5
:queues:
  - [low, 1]
  - [default, 1]

Deployment Commands

When using per-server configurations, Capistrano will:

• Install the appropriate services on each server during cap production sidekiq:install
• Start only the configured services on each server during cap production sidekiq:start
• Manage each server's services independently

You can also target specific servers:

cap production sidekiq:restart --hosts=worker1.example.com

Multiple Processes

There are two ways to run multiple Sidekiq processes:

Option 1: Multiple Config Files (Recommended)

Create separate config files for different workloads:

# config/sidekiq_critical.yml
:concurrency: 5
:queues:
  - [critical, 2]

# config/sidekiq_default.yml  
:concurrency: 10
:queues:
  - [default, 1]
  - [low, 1]

Configure in deploy.rb:

set :sidekiq_config_files, ['sidekiq_critical.yml', 'sidekiq_default.yml']

This creates separate services:

• myapp_sidekiq_production.sidekiq_critical
• myapp_sidekiq_production.sidekiq_default

Benefits:

• Clear purpose for each process
• Independent scaling and management
• Different configurations per process
• Meaningful service names

Option 2: Sidekiq Enterprise (Sidekiqswarm)

For simple scaling with identical processes:

set :sidekiq_command, 'sidekiqswarm'
set :sidekiq_service_unit_env_vars, ['SIDEKIQ_COUNT=10']

This runs 10 identical processes with a single service.

Quick Setup for Multiple Processes

Use the helper task to generate config files:

# Generate 5 config files
bundle exec cap production sidekiq:helpers:generate_configs[5]

# This creates:
# - config/sidekiq.yml (critical, high queues)
# - config/sidekiq_1.yml (default, medium queues)
# - config/sidekiq_2.yml (low, background queues)
# - config/sidekiq_3.yml (low, background queues)
# - config/sidekiq_4.yml (low, background queues)

Then add to your deploy.rb:

set :sidekiq_config_files, ['sidekiq.yml', 'sidekiq_1.yml', 'sidekiq_2.yml', 'sidekiq_3.yml', 'sidekiq_4.yml']

Available Tasks

# View all available tasks
cap -T sidekiq

# Common commands
cap sidekiq:start              # Start Sidekiq
cap sidekiq:stop               # Stop Sidekiq (graceful shutdown)
cap sidekiq:restart            # Restart Sidekiq
cap sidekiq:quiet              # Quiet Sidekiq (stop processing new jobs)
cap sidekiq:install            # Install Sidekiq systemd service
cap sidekiq:uninstall          # Remove Sidekiq systemd service
cap sidekiq:enable             # Enable Sidekiq systemd service
cap sidekiq:disable            # Disable Sidekiq systemd service
cap sidekiq:mark_deploy        # Mark deployment in Sidekiq metrics (Sidekiq 7+)

# Helper tasks
cap sidekiq:helpers:generate_configs[3]  # Generate 3 config files
cap sidekiq:helpers:show_config          # Show current configuration

Systemd Integration

For detailed information about systemd integration, see Systemd Integration Guide.

Deployment Tracking (Sidekiq 7+)

Sidekiq 7 introduced metrics with deployment tracking. Enable it to see deployment markers in your Sidekiq metrics graphs:

Configuration

# config/deploy.rb

# Enable deployment marking
set :sidekiq_mark_deploy, true

# Optional: Custom deploy label (defaults to git commit description)
set :sidekiq_deploy_label, "v#{fetch(:current_revision)[0..6]} deployed to #{fetch(:stage)}"

How It Works

When enabled, the gem will:

• Automatically mark deployments after successful deploys
• Show vertical red lines in Sidekiq's metrics graphs
• Display the deploy label when hovering over the marker

This helps correlate performance changes with deployments.

Sidekiq Enterprise (Swarm) Support

This gem supports Sidekiq Enterprise's sidekiqswarm command for managing multiple processes with a single service.

Configuration

# config/deploy.rb

# Use sidekiqswarm instead of sidekiq
set :sidekiq_command, 'sidekiqswarm'

# Configure swarm-specific environment variables
set :sidekiq_service_unit_env_vars, [
  'SIDEKIQ_COUNT=10',          # Number of processes to spawn
  'SIDEKIQ_MAXMEM_MB=512',     # Memory limit per process
  'SIDEKIQ_PRELOAD_APP=1',     # Preload Rails app for memory efficiency
  'RAILS_ENV=production'
]

# Optional: Configure shutdown timeout for graceful stops
set :sidekiq_command_args, -> { "-e #{fetch(:sidekiq_env)} -t 30" }

Benefits of Sidekiqswarm

• Single Service: Manage N processes with one systemd service
• Memory Efficiency: Preload app once, fork processes
• Auto-restart: Processes that exceed memory limits are automatically restarted
• Simplified Management: No need for multiple service files

Per-Server Swarm Configuration

server 'worker1.example.com',
  roles: [:worker],
  sidekiq_service_unit_env_vars: ['SIDEKIQ_COUNT=20', 'SIDEKIQ_MAXMEM_MB=1024']

server 'worker2.example.com',
  roles: [:worker],
  sidekiq_service_unit_env_vars: ['SIDEKIQ_COUNT=5', 'SIDEKIQ_MAXMEM_MB=512']

Working with Systemd Logs

View Sidekiq service logs using journalctl:

# View last 100 lines of logs
journalctl -u sidekiq_myapp_production -n 100

# Follow logs in real-time
journalctl -u sidekiq_myapp_production -f

Log File Configuration

Modern Systemd (v240+, e.g., Ubuntu 20.04+)

Log files are configured automatically using the append: functionality in the systemd service file.

Legacy Systemd Systems

For systems with older Systemd versions where append: is not supported:

• Sidekiq messages are sent to syslog by default
• Configure system logger to filter Sidekiq messages

Example Application

A complete example application demonstrating the usage of this gem is available at: https://github.com/seuros/capistrano-example-app

Contributing

• Fork the repository
• Create your feature branch (git checkout -b my-new-feature)
• Commit your changes (git commit -am 'Add some feature')
• Push to the branch (git push origin my-new-feature)
• Create new Pull Request