sigdump

sigdump

In short: SIGQUIT of Java VM for Ruby.

Server applications (like Rails apps) cause performance problems, deadlock or memory swapping from time to time. But it's difficult to reproduce such kind of problems. sigdump makes it possible to get information from a running process without restarting. Just sending SIGCONT signal will dump backtrace and memory profile to /tmp/sigdump-<pid>.log file.

sigdump dumps following information (see also Sample output):

• Backtrace of all threads
• Number of allocated objects per class
• GC profiler reports if GC profiler is enabled (GC::Profiler.enable is called)
• Stacktrace of Java threads for each Ruby threads if the runtime is JRuby

Install

Just install sigdump gem and add require 'sigdump/setup' line to your code. Or you can use Bundler as following:

# Gemfile
gem 'sigdump', require: 'sigdump/setup'

Note for Resque:

You need to change the default signal (SIGCONT) because Rescue traps SIGCONT and it conflicts with sigdump. To change the signal, set name of a signal to SIGDUMP_SIGNAL environment variable. For Rails, you can add following lines to environment.rb file:

# setup sigdump: https://github.com/frsyuki/sigdump
ENV['SIGDUMP_SIGNAL'] = 'TSTP'
require 'sigdump/setup'

Usage

Send SIGCONT signal to the Ruby process. It dumps backtrace and memory profile to /tmp/sigdump-<pid>.log file.

$ kill -CONT <pid>
$ cat /tmp/sigdump-<pid>.log

Set SIGDUMP_SIGNAL environment variable to change the signal (default: SIGCONT).

Set SIGDUMP_PATH environment variable to change the output path (default: /tmp/sigdump-<pid>.log). You can set "-" here to dump to STDOUT, or "+" to STDERR.

Sample outout

$ cat /tmp/sigdump-9218.log
Sigdump at 2013-04-24 16:57:12 +0000 process 9218 (unicorn worker[3] -E staging -c /etc/unicorn/staging.rb -E staging)
  Thread #<Thread:0x00000001424518> status=run priority=0
      /srv/staging/current/vendor/bundle/ruby/1.9.1/gems/sigdump-0.1.0/lib/sigdump.rb:32:in `dump_backtrace'
      /srv/staging/current/vendor/bundle/ruby/1.9.1/gems/sigdump-0.1.0/lib/sigdump.rb:19:in `block in dump_all_thread_backtrace'
      /srv/staging/current/vendor/bundle/ruby/1.9.1/gems/sigdump-0.1.0/lib/sigdump.rb:18:in `each'
      /srv/staging/current/vendor/bundle/ruby/1.9.1/gems/sigdump-0.1.0/lib/sigdump.rb:18:in `dump_all_thread_backtrace'
      /srv/staging/current/vendor/bundle/ruby/1.9.1/gems/sigdump-0.1.0/lib/sigdump.rb:9:in `block (2 levels) in install_thread_dump_handler'
      /srv/staging/current/vendor/bundle/ruby/1.9.1/gems/sigdump-0.1.0/lib/sigdump.rb:91:in `open'
      /srv/staging/current/vendor/bundle/ruby/1.9.1/gems/sigdump-0.1.0/lib/sigdump.rb:91:in `_open_dump_path'
      /srv/staging/current/vendor/bundle/ruby/1.9.1/gems/sigdump-0.1.0/lib/sigdump.rb:7:in `block in install_thread_dump_handler'
      /srv/staging/current/vendor/bundle/ruby/1.9.1/gems/unicorn-4.3.1/lib/unicorn/http_server.rb:626:in `call'
      /srv/staging/current/vendor/bundle/ruby/1.9.1/gems/unicorn-4.3.1/lib/unicorn/http_server.rb:626:in `select'
      /srv/staging/current/vendor/bundle/ruby/1.9.1/gems/unicorn-4.3.1/lib/unicorn/http_server.rb:626:in `worker_loop'
      /srv/staging/current/vendor/bundle/ruby/1.9.1/gems/unicorn-4.3.1/lib/unicorn/http_server.rb:487:in `spawn_missing_workers'
      /srv/staging/current/vendor/bundle/ruby/1.9.1/gems/unicorn-4.3.1/lib/unicorn/http_server.rb:137:in `start'
      /srv/staging/current/vendor/bundle/ruby/1.9.1/gems/unicorn-4.3.1/bin/unicorn:121:in `<top (required)>'
      /srv/staging/current/vendor/bundle/ruby/1.9.1/bin/unicorn:23:in `load'
      /srv/staging/current/vendor/bundle/ruby/1.9.1/bin/unicorn:23:in `<main>'
  GC stat:
      count: 34
      heap_allocated_pages: 1366
      heap_sorted_length: 1368
      heap_allocatable_pages: 0
      heap_available_slots: 556777
      heap_live_slots: 551708
      heap_free_slots: 5069
      heap_final_slots: 0
      heap_marked_slots: 363350
      heap_swept_slots: 58807
      heap_eden_pages: 1366
      heap_tomb_pages: 0
      total_allocated_pages: 1367
      total_freed_pages: 1
      total_allocated_objects: 2438499
      total_freed_objects: 1886791
      malloc_increase_bytes: 650416
      malloc_increase_bytes_limit: 16777216
      minor_gc_count: 25
      major_gc_count: 9
      remembered_wb_unprotected_objects: 5122
      remembered_wb_unprotected_objects_limit: 5222
      old_objects: 348964
  Built-in objects:
   367,492: TOTAL
   208,193: T_STRING
    61,817: T_ARRAY
    37,343: T_DATA
    28,293: T_NODE
    10,678: T_OBJECT
     6,385: T_HASH
     5,957: T_CLASS
     2,300: T_ICLASS
     2,184: T_REGEXP
     1,547: T_MODULE
       900: T_FLOAT
       677: T_STRUCT
       497: T_BIGNUM
       432: T_MATCH
       251: T_RATIONAL
        29: T_FILE
         8: FREE
         1: T_COMPLEX
  All objects:
   207,335: String
    32,987: Array
    28,665: RubyVM::InstructionSequence
     5,863: Hash
     3,759: RubyVM::Env
     3,680: Proc
     2,338: Class
     2,184: Regexp
     1,632: MIME::Type
     1,547: Module
     1,040: Gem::Version
       982: Gem::Requirement
       945: Float
       920: Journey::Nodes::Cat
       804: Time
       660: Gem::Dependency
       497: Bignum
  ...
  String 7,556,137 bytes
   Array 821 elements
    Hash 90 pairs