github.com/yinonavraham/go-profiling-demo

Go Profiling Demo

Go application for demoing profiling in Go using standard library tools and packages such as pprof, trace and expvar.

This demo was presented in a talk in a Go meetup hosted by AppsFlyer and JFrog on Oct. 19th 2021. The recording (in Hebrew) is available on YouTube. The slides used together with this demo are also available: Web Application Profiling 101 - Slides (pdf)

This is a simple HTTP server application, serving files from a data directory. It mainly implements the following endpoint:

GET /file/<file-path>

Setup

• Clone this repository
• Install wrk benchmarking tool
  (used to send concurrent requests to the demo server application)

Demo Flow

Following are the steps of this demo. Follow the links to each step to read what is done in the step and to see the code changes.

• Step 0 - Demo preparation and first run
• Step 1 - Add the pprof endpoints
• Step 2 - Collect CPU profile and visualize it
• Step 3 - Use execution tracer to visualize Go routines scheduling, GC events, and more
• Step 4 - Collect memory profile and visualize it
• Step 5 - Improvement attempt #1
• Step 6 - Improvement attempt #2
• Step 7 - Improvement attempt #3
• Step 8 - Add a custom profile
• Step 9 - Expose operational values

Notes

Don't use the default serve mux

Using the default serve mux complicates the ability to put access control and enables other packages to expose endpoints implicitly, similar to how the pprof endpoints are added just by adding an anonymous import. It is advised to use your own serve mux and add the pprof endpoints explicitly, preferably with an auth middleware. For example:

func main() {
   mux := http.NewServeMux()
   mux.HandleFunc("/file/", handleGetFile)
   addPprofHandlers(mux)
   // ...
	log.Fatal(http.ListenAndServe(address, authMiddleware(mux))) 
}

func addPprofHandlers(mux *http.ServeMux) {
	mux.HandleFunc("/debug/pprof/", pprof.Index)
	mux.HandleFunc("/debug/pprof/cmdline", pprof.Cmdline)
	mux.HandleFunc("/debug/pprof/symbol", pprof.Symbol)
	mux.HandleFunc("/debug/pprof/profile", pprof.Profile)
	mux.HandleFunc("/debug/pprof/trace", pprof.Trace)
}

func authMiddleware(next http.Handler) http.Handler {
   // ...
}

Links

• Go Diagnostics (GoLang Documentation)
• pprof tool by Google
• Profiling Go Programs (The Go Blog)