I am trying to find performance bottlenecks by using the perf tool on a kubernetes pod. I have already set the following on the instance hosting the pod:
"kernel.kptr_restrict" = "0"
"kernel.perf_event_paranoid" = "0"
However, I have to problems.
When I collect samples through
perf record -a -F 99 -g -p <PID> --call-graph dwarfand feed it to speedscore or similarly to a flamegraph, I still see question marks???and the process that I would like to see its CPU usage breakdown (C++ based), the aforementioned???is on the top of the stack and system calls fall below it. The main process is the one that has???around on it.I tried running
perf topand it says
Failed to mmap with 1 (Operation not permitted)
My questions are:
- For collecting perf top, what permissions do I need to change on the host instance of the pod?
- Which other settings do I need to change at the instance level so I don't see any more
???showing up on perf's output. I would like to see the function call stack of the process, not just the system calls. See the following stack:
The host OS is ubuntu.
Zooming in on the first system call, you would see this, but this only gives me a fraction of the CPU time spent and only the system calls.
UPDATE:
I was able to run perf top, by setting
"kernel.perf_event_paranoid" = "-1". However, as seen in the image below, the process I'm trying to profile (I've blackened out the name to hide the name), is not showing me function names but just addresses. I try running them through addr2line, but it says addr2line: 'a.out': No such file.
How can I get the addresses resolve to function names on the pod? Is it even possible?
I was also able to fix the memory-function mapping with perf top. This was due to the fact that I was trying to run perf from a different container than where the process was running (same pod, different container). There may be a way to add extra information, but just moving the perf to the container running the process fixed it.
