github.com/pyroscope-io/otel-profiling-go

Profiling Instrumentation

NOTE: This is an experimental package -- and will be officially supported in future versions of Pyroscope

The package provides means to integrate tracing with profiling. More specifically, a TracerProvider implementation, that annotates profiling data with span IDs: when a new trace span emerges, the tracer adds a profile_id pprof tag that points to the span. This makes it possible to filter out a profile of a particular trace span in Pyroscope.

Jaeger Example

You can find a full Jaeger example (with custom Jaeger UI) in the tracing/jaeger folder in the Pyroscope repository.

Baseline Diffs

We also added functionality where each individual span is compared to a baseline of spans with similar properties and the diff can be shown in the UI:

Grafana Example

For another example of what this package allows you to do you can see with Grafana the ability to link between logs, traces and profiles in the following video (source):

Other Notes

Note that the module does not control pprof profiler itself – it still needs to be started for profiles to be collected. This can be done either via runtime/pprof package, or using the Pyroscope client.

By default, only the root span gets annotated (the first span created locally), this is done to circumvent the fact that the profiler records only the time spent on CPU. Otherwise, all the children profiles should be merged to get the full representation of the root span profile.

There are few limitations:

• Only Go CPU profiling is fully supported at the moment.
• Due to the very idea of the sampling profilers, spans shorter than the sample interval may not be captured. For example, Go CPU profiler probes stack traces 100 times per second, meaning that spans shorter than 10ms may not be captured.