3

I like the fp method for collecting call stacks with perf record since it's lightweight and less complex than dwarf. However, when I look at the call stacks/flamegraphs I get when a program uses the C++ standard library, they are not correct.

Here is a test program:

#include <algorithm>
#include <iomanip>
#include <iostream>
#include <sstream>
#include <string>
#include <vector>

int __attribute__((noinline)) stupid_factorial(int x) {
    std::vector<std::string> xs;
    // Need to convert numbers to strings or it will all get inlined
    for (int i = 0; i < x; ++i) {
        std::stringstream ss;
        ss << std::setw(4) << std::setfill('0') << i;
        xs.push_back(ss.str());
    }
    int res = 1;
    while(std::next_permutation(xs.begin(), xs.end())) {
        res += 1;
    };
    return res;
}

int main() {
    std::cout << stupid_factorial(11) << "\n";
}

And here is the flame graph:

enter image description here

It was generated by the following steps on Ubuntu 20.04 in a Docker container:

g++ -Wall -O3 -g -fno-omit-frame-pointer program.cpp -o 6_stl.bin
# Make sure you have libc6-prof and libstdc++6-9-dbg installed
env LD_LIBRARY_PATH=/lib/libc6-prof/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu/debug:${LD_LIBRARY_PATH} perf record -F 1000 --call-graph fp -- ./6_stl.bin
# Make sure you have https://github.com/jonhoo/inferno installed
perf script | inferno-collapse-perf | inferno-flamegraph > flamegraph.svg

The main thing that's wrong with this is that not all functions are children of stupid_factorial, e.g. __memcmp_avx2_movbe. With dwarf, they are. In more complex programs, I have even seen functions like these being outside main. __dynamic_cast for instance is one that often has no parent.

In gdb, I always see correct backtraces, including for the functions that do not appear correctly here. Is it possible to get correct fp call stacks with libstdc++ without compiling it myself (which seems like a lot of work)?

There are also other oddities, though I couldn't reproduce them in Ubuntu 18.04 (outside the Docker container):

  • There is an unresolved function in libstdc++.so.6.28.
  • There is an unresolved function in my own binary, 6_stl.bin, on the very left. This is also the case with dwarf.
nnnmmm
  • 7,964
  • 4
  • 22
  • 41
  • You'd probably need a build of `libstdc++` compiled with `-fno-omit-frame-pointer`, although I would have expected that `__memcmp_avx2_movbe` is called directly (at least after lazy dynamic linking is done on the first call, if you didn't use `-fno-plt`). GDB uses dwarf (`.eh_frame`) backtracing, not frame-pointers, unless the file is missing stack-unwind dwarf metadata – Peter Cordes Jul 05 '21 at 17:22
  • Have you tried to compile with `-O0` and see the difference? – prehistoricpenguin Jul 06 '21 at 02:34
  • @PeterCordes I was afraid that I'd need to do such a thing. `__memcmp_avx2_movbe` being called outside `stupid_factorial` is not expected to me since it wouldn't take so much time if it came from `std::cout`, and `stupid_factorial` isn't inlined. – nnnmmm Jul 06 '21 at 12:07
  • 1
    @prehistoricpenguin The flamegraph doesn't look skewed without `-O3`, but unfortunately I'm only interested in the performance of the optimized code. – nnnmmm Jul 06 '21 at 12:09
  • @PeterCordes I tried compiling libstdc++ with `-fno-omit-frame-pointer`, and while it helps with some problems (like the unresolved `[6_stl.bin]` symbol), it doesn't solve the primary issue I described. – nnnmmm Jul 13 '21 at 13:11

2 Answers2

4

With your code, 20.04 x86_64 ubuntu, perf record --call-graph fp with and without -e cycles:u I have similar flamegraph as viewed with https://speedscope.app (prepare data with perf script > out.txt and select out.txt in the webapp).

Is it possible to get correct fp call stacks with libstdc++ without compiling it myself (which seems like a lot of work)?

No, call-graph method 'fp' is implemented in linux kernel code in very simple way: https://elixir.bootlin.com/linux/v5.4/C/ident/perf_callchain_user - https://elixir.bootlin.com/linux/v5.4/source/arch/x86/events/core.c#L2464

perf_callchain_user(struct perf_callchain_entry_ctx *entry, struct pt_regs *regs)
{ 
    ...
    fp = (unsigned long __user *)regs->bp;
    perf_callchain_store(entry, regs->ip);
    ...
    // where max_stack is probably around 127 = PERF_MAX_STACK_DEPTH     https://elixir.bootlin.com/linux/v5.4/source/include/uapi/linux/perf_event.h#L1021
    while (entry->nr < entry->max_stack) {
        ...
        if (!valid_user_frame(fp, sizeof(frame)))
            break;
        bytes = __copy_from_user_nmi(&frame.next_frame, fp, sizeof(*fp));
        bytes = __copy_from_user_nmi(&frame.return_address, fp + 1, sizeof(*fp));

        perf_callchain_store(entry, frame.return_address);
        fp = (void __user *)frame.next_frame;
    }
}

It can't find correct frames for -fomit-frame-pointer compiled code.

For incorrect call stacks with main -> __memcmp_avx2_movbe there is only call stack data generated by kernel in perf.data file, no copy of user stack fragment, no register data:

setarch x86_64 -R env LD_LIBRARY_PATH=/lib/libc6-prof/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu/debug:${LD_LIBRARY_PATH} perf record -F 1000 --call-graph fp  -- ./6_stl.bin
perf script -D | less

869122666352078 0xae0 [0x58]: PERF_RECORD_SAMPLE(IP, 0x4002): 12267/12267: 0x7ffff7d51670 period: 2332683 addr: 0
... FP chain: nr:5
.....  0: fffffffffffffe00
.....  1: 00007ffff7d51670
.....  2: 0000555555556452
.....  3: 00007ffff7be90fb
.....  4: 00005555555564de
 ... thread: 6_stl.bin:12267
 ...... dso: /usr/lib/libc6-prof/x86_64-linux-gnu/libc-2.31.so
6_stl.bin 12267 869122.666352:    2332683 cycles: 
            7ffff7d51670 __memcmp_avx2_movbe+0x140 (/usr/lib/libc6-prof/x86_64-linux-gnu/libc-2.31.so)
            555555556452 main+0x12 (/home/user/so/68259699/6_stl.bin)
            7ffff7be90fb __libc_start_main+0x10b (/usr/lib/libc6-prof/x86_64-linux-gnu/libc-2.31.so)
            5555555564de _start+0x2e (/home/user/so/68259699/6_stl.bin)

So, with this method user-space perf tool can't use any additional information to fix the call stack. With dwarf method there are registers and partial dump of user stack data on every sample event.

Gdb has full access to live process and can use any information, all registers, read any amount of user process stack, read additional debug info for program and libraries. And doing advanced and slow backtrace in gdb is not limited by time or security or uninterruptible context. Linux kernel should record perf sample in small time, it can't access swapped data or debug sections or debug info files, it should not do complex parsing (which can have some bugs).

Debug version of libstdc++ may help (sudo apt install libstdc++6-9-dbg), but it is slow. And it did not help me to find lost backtrace of this asm-implemented __memcmp_avx2_movbe (libc: sysdeps/x86_64/multiarch/memcmp-avx2-movbe.S)

If you want full backtrace, I think you should find how to recompile a world (or only all libraries used by your target application). Probably it will be easier not with Ubuntu but with something like gentoo or arch or apline?

If you are interested only in performance why do you want the flamegraph? Flat profile will catch most performance data; non-ideal flamegraph can be useful too.

osgx
  • 90,338
  • 53
  • 357
  • 513
2

When you look at the source code for the __memcmp_avx2_movbe function, you see that it doesn't have a function prologue.

Therefore, we should expect the immediate parent frame of __memcmp_avx2_movbe to be skipped in the backtrace. The innermost frame will still be correctly identified as __memcmp_avx2_movbe from the instruction pointer, but the return address on the stack that is identified by the frame pointer will belong to the grandparent.

When the stupid_factorial function is the parent of __memcmp_avx2_movbe (because all intermediate functions between those two are inlined), that could explain the primary issue from the question. The other issues are resolved by using a libstdc++ compiled with frame pointers as described here.

nnnmmm
  • 7,964
  • 4
  • 22
  • 41