grpc-native-debug

This package contains debug symbols that can be useful for debugging applications that use grpc pre-compiled binary gems.

An example of a pre-compiled binary gem is grpc-1.58.0-x86_64-linux.gem (as opposed to a source-built gem like grpc-1.58.0.gem).

grpc-native-debug gems contain debug symbols which complement the native libraries in these grpc binary gems. After fetching and unpacking a proper grpc-native-debug gem, one can load the correct .dbg symbol file to debug their grpc application.

Background

grpc-ruby pre-compiled binary gems are released with debug symbols stripped. As a consequence, if you are to examine a grpc stack trace in a debugger for example, a lot of information will initially be missing.

Using grpc-native-debug

Finding the correct grpc-native-debug gem

Each grpc-native-debug gem is one-to-one with a grpc gem. Specifically:

• The version of a grpc-native-debug gem must match the version of the grpc gem.

• The ruby platform of a grpc-native-debug gem must match the ruby platform of the grpc gem.

So for example, if you are debugging grpc-1.60.1-x86_64-linux.gem, then you need to fetch grpc-native-debug-1.60.1-x86_64-linux.gem.

Finding the correct .dbg symbol file

Each grpc-native-debug gem has a top-level symbols directory containing symbol files ending in .dbg.

grpc binary gems are shipped with multiple native libraries. There is one native library for each supported minor version of ruby. As such, grpc-native-debug gems have exactly one .dbg file for each native library in the corresponding grpc gem.

If you unpack a grpc-native-debug gem and look at the symbols directory, you might see something like this:

grpc-native-debug-1.60.1-x86_64-linux/symbols/grpc-1.60.1-x86_64-linux-ruby-3.0.dbg
grpc-native-debug-1.60.1-x86_64-linux/symbols/grpc-1.60.1-x86_64-linux-ruby-2.7.dbg
grpc-native-debug-1.60.1-x86_64-linux/symbols/grpc-1.60.1-x86_64-linux-ruby-3.2.dbg
grpc-native-debug-1.60.1-x86_64-linux/symbols/grpc-1.60.1-x86_64-linux-ruby-3.1.dbg

In each of these .dbg files, the ruby-<major>-<minor> portion of the string indicates which ruby version it's supposed to be used with.

So for example, if you are debugging grpc-1.60.1-x86_64-linux.gem on ruby-3.0, then you need to use symbol file grpc-native-debug-1.60.1-x86_64-linux/symbols/grpc-1.60.1-x86_64-linux-ruby-3.0.dbg.

Putting symbols into action (example gdb workflow)

There are a variety of ways to use these symbols.

As a toy example, suppose we are running an application under gdb using:

• ruby-3.0

• grpc-1.60.1.x86_64-linux.gem

At first, in gdb we might dump a grpc-ruby stack trace looking something like this:

(gdb) bt
#0  0x00007ffff7926e56 in epoll_wait (epfd=5, events=0x7ffff3cb4144, maxevents=100, timeout=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
#1  0x00007ffff383eb9e in ?? () from /home/.rvm/gems/ruby-3.0.0/gems/grpc-1.60.1-x86_64-linux/src/ruby/lib/grpc/3.0/grpc_c.so
#2  0x00007ffff355e002 in ?? () from /home/.rvm/gems/ruby-3.0.0/gems/grpc-1.60.1-x86_64-linux/src/ruby/lib/grpc/3.0/grpc_c.so
#3  0x00007ffff38466e2 in ?? () from /home/.rvm/gems/ruby-3.0.0/gems/grpc-1.60.1-x86_64-linux/src/ruby/lib/grpc/3.0/grpc_c.so
#4  0x00007ffff35ba2ea in ?? () from /home/.rvm/gems/ruby-3.0.0/gems/grpc-1.60.1-x86_64-linux/src/ruby/lib/grpc/3.0/grpc_c.so
#5  0x00007ffff34abf6b in ?? () from /home/.rvm/gems/ruby-3.0.0/gems/grpc-1.60.1-x86_64-linux/src/ruby/lib/grpc/3.0/grpc_c.so
#6  0x00007ffff7c67ca7 in rb_nogvl (func=0x7ffff34abed3, data1=0x0, ubf=<optimized out>, data2=<optimized out>, flags=<optimized out>) at thread.c:1669
#7  0x00007ffff34ab110 in ?? () from /home/.rvm/gems/ruby-3.0.0/gems/grpc-1.60.1-x86_64-linux/src/ruby/lib/grpc/3.0/grpc_c.so
#8  0x00007ffff7c6780c in thread_do_start (th=0x555555ad16e0) at thread.c:769
#9  thread_start_func_2 (th=th@entry=0x555555ad16e0, stack_start=<optimized out>) at thread.c:822
#10 0x00007ffff7c679a6 in thread_start_func_1 (th_ptr=<optimized out>) at /home/.rvm/src/ruby-3.0.0/thread_pthread.c:994
#11 0x00007ffff78a63ec in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:444
#12 0x00007ffff7926a4c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

We could take the following steps to get more debug info:

1) Fetch the correct grpc-native-debug gem

cd /home
gem fetch grpc-native-debug-1.60.1.x86_64-linux.gem
gem unpack grpc-native-debug-1.60.1.x86_64-linux.gem

(note again the version and platform of grpc-native-debug must match the grpc gem)

2) Load debug symbols (for ruby-3.0)

(gdb) info sharedlibrary
From                To                  Syms Read   Shared Object Library
...
0x00007ffff3497450  0x00007ffff3a61912  Yes (*)     /home/.rvm/gems/ruby-3.0.0/gems/grpc-1.60.1-x86_64-linux/src/ruby/lib/grpc/3.0/grpc_c.so
0x00007ffff3e78730  0x00007ffff3ea60df  Yes (*)     /home/.rvm/gems/ruby-3.0.0/gems/google-protobuf-3.24.4-x86_64-linux/lib/google/3.0/protobuf_c.so
(*): Shared library is missing debugging information.
(gdb) add-symbol-file /home/grpc-native-debug-1.60.1-x86_64-linux/symbols/grpc-1.60.1-x86_64-linux-ruby-3.0.dbg 0x00007ffff3497450
add symbol table from file "/home/grpc-native-debug-1.60.1-x86_64-linux/symbols/grpc-1.60.1-x86_64-linux-ruby-3.0.dbg" at
	.text_addr = 0x7ffff3497450
(y or n) y
Reading symbols from /home/grpc-native-debug-1.60.1-x86_64-linux/symbols/grpc-1.60.1-x86_64-linux-ruby-3.0.dbg...
(gdb)

Our stack trace might look more like this now:

(gdb) bt
#0  0x00007ffff7926e56 in epoll_wait (epfd=5, events=0x7ffff3cb4144, maxevents=100, timeout=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
#1  0x00007ffff383eb9e in do_epoll_wait (ps=0x555555ad1690, deadline=...) at src/core/lib/iomgr/ev_epoll1_linux.cc:723
#2  pollset_work (ps=0x555555ad1690, worker_hdl=0x0, deadline=...) at src/core/lib/iomgr/ev_epoll1_linux.cc:1038
#3  0x00007ffff355e002 in pollset_work (pollset=<optimized out>, worker=<optimized out>, deadline=...) at src/core/lib/iomgr/ev_posix.cc:249
#4  0x00007ffff38466e2 in grpc_pollset_work (pollset=<optimized out>, worker=<optimized out>, deadline=...) at src/core/lib/iomgr/pollset.cc:48
#5  0x00007ffff35ba2ea in cq_next (cq=0x555555ad1510, deadline=..., reserved=<optimized out>) at src/core/lib/surface/completion_queue.cc:1043
#6  0x00007ffff34abf6b in run_poll_channels_loop_no_gil (arg=arg@entry=0x0) at ../../../../src/ruby/ext/grpc/rb_channel.c:663
#7  0x00007ffff7c67ca7 in rb_nogvl (func=0x7ffff34abed3 <run_poll_channels_loop_no_gil>, data1=0x0, ubf=<optimized out>, data2=<optimized out>, flags=flags@entry=0) at thread.c:1669
#8  0x00007ffff7c68138 in rb_thread_call_without_gvl (func=<optimized out>, data1=<optimized out>, ubf=<optimized out>, data2=<optimized out>) at thread.c:1785
#9  0x00007ffff34ab110 in run_poll_channels_loop (arg=<optimized out>) at ../../../../src/ruby/ext/grpc/rb_channel.c:734
#10 0x00007ffff7c6780c in thread_do_start (th=0x555555ad16e0) at thread.c:769
#11 thread_start_func_2 (th=th@entry=0x555555ad16e0, stack_start=<optimized out>) at thread.c:822
#12 0x00007ffff7c679a6 in thread_start_func_1 (th_ptr=<optimized out>) at /home/.rvm/src/ruby-3.0.0/thread_pthread.c:994
#13 0x00007ffff78a63ec in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:444
#14 0x00007ffff7926a4c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

This is better, but if we try to examine a frame closely we'll notice that source file information is still missing:

(gdb) up
#1  0x00007ffff383eb9e in do_epoll_wait (ps=0x555555ad1690, deadline=...) at src/core/lib/iomgr/ev_epoll1_linux.cc:723
723	src/core/lib/iomgr/ev_epoll1_linux.cc: No such file or directory.
(gdb) list
718	in src/core/lib/iomgr/ev_epoll1_linux.cc
(gdb)

3) Resolve source files

First, we fetch the source grpc gem at the exact same version of our binary grpc gem:

cd /home
gem fetch grpc-1.60.1.gem
gem unpack grpc-1.60.1.gem

Now we can load those sources in gdb:

(gdb) dir /home/grpc-1.60.1
Source directories searched: /home/grpc-1.60.1:$cdir:$cwd
(gdb)

Our stack frame will might look more like this now:

(gdb) list
warning: Source file is more recent than executable.
718	  int timeout = poll_deadline_to_millis_timeout(deadline);
719	  if (timeout != 0) {
720	    GRPC_SCHEDULING_START_BLOCKING_REGION;
721	  }
722	  do {
723	    r = epoll_wait(g_epoll_set.epfd, g_epoll_set.events, MAX_EPOLL_EVENTS,
724	                   timeout);
725	  } while (r < 0 && errno == EINTR);
726	  if (timeout != 0) {
727	    GRPC_SCHEDULING_END_BLOCKING_REGION;
(gdb)

But if we move up a few stack frames we might still be missing some source information:

(gdb) up
#6  0x00007ffff34abf6b in run_poll_channels_loop_no_gil (arg=arg@entry=0x0) at ../../../../src/ruby/ext/grpc/rb_channel.c:663
663	../../../../src/ruby/ext/grpc/rb_channel.c: No such file or directory.

A portion of the grpc-ruby native extension is built from a sub-directory: src/ruby/ext/grpc. So we also need to add that sub-directory, to fix this:

(gdb) dir /home/grpc-1.60.1/src/ruby/ext/grpc
Source directories searched: /home/grpc-1.60.1/src/ruby/ext/grpc:/home/grpc-1.60.1:$cdir:$cwd

Note the additional info:

(gdb) list
warning: Source file is more recent than executable.
658	  gpr_mu_lock(&global_connection_polling_mu);
659	  gpr_cv_broadcast(&global_connection_polling_cv);
660	  gpr_mu_unlock(&global_connection_polling_mu);
661
662	  for (;;) {
663	    event = grpc_completion_queue_next(
664	        g_channel_polling_cq, gpr_inf_future(GPR_CLOCK_REALTIME), NULL);
665	    if (event.type == GRPC_QUEUE_SHUTDOWN) {
666	      break;
667	    }
(gdb)

Support

grpc-native-debug currently only supports:

• ruby platforms: x86_64-linux and x86-linux

• grpc >= 1.60.0