Profile CPU + I/O

[Yamakaky]

By default, pprof only counts the CPU usage of a trace and not the I/O time. In my case, I have multiple microservices that spend the majority of their time calling themselves and the database, so the CPU flamegraph is of limited use. For example, one API endpoint takes 200ms to call another service, but only 0-1 samples are recorded in pyroscope since so little time is spent in CPU. This means the link for the opentelemetry trace is empty most of the time.

I see https://github.com/felixge/fgprof seems to do what I need. It is implemented as a standalone HTTP handler, so this would need extracting the relevant code from https://github.com/felixge/fgprof/blob/master/fgprof.go, or working with them to split the handler from the business code.

I could help for that.

[kolesnikovae]

Yeah, CPU wall time profile can be extremely useful in certain cases.

Are you using pull mode or Go client (push)? As for supporting fgprof pprof endpoint – I think pyroscope server should be able to pull data, but this will require some extra configuration (for example, grafana/pyroscope#768 (comment)) – I'll give it a try this week and get back with results

I'm not 100% sure we want to add a dependency to our Go client but we will discuss it internally

[Yamakaky]

I'm using push, I use a centralized server for multiple dynamic instances so pull mode is complicated.

https://www.brendangregg.com/offcpuanalysis.html indicates cpu wall time can have a noticeable performance impact, si maybe add it as an option?