37

When using perf report, I don't see any symbols for my program, instead I get output like this: 

$ perf record /path/to/racket ints.rkt 10000
$ perf report --stdio

# Overhead   Command      Shared Object  Symbol
# ........  ........  .................  ......
#
    70.06%  ints.rkt  [unknown]          [.] 0x5f99b8        
    26.28%  ints.rkt  [kernel.kallsyms]  [k] 0xffffffff8103d0ca
     3.66%  ints.rkt  perf-32046.map     [.] 0x7f1d9be46650

Which is fairly uninformative.

The relevant program is built with debugging symbols, and the sysprof tool shows the appropriate symbols, as does Zoom, which I think is using perf under the hood.

Note that this is on x86-64, so the binary is compiled with -fomit-frame-pointer, but that's the case when running under the other tools as well.

Sam Tobin-Hochstadt
  • 4,983
  • 1
  • 21
  • 43
  • And having tried it with `-fno-omit-frame-pointer`, that has no effect. – Sam Tobin-Hochstadt Jun 07 '12 at 19:59
  • Can you create a minimal example which exhibits the problem? Start with a basic binary which doesn't and bisect the differences. – Brian Cain Jun 23 '12 at 14:55
  • @BrianCain, the relevant binary is quite large, and thus bisecting isn't really feasible. I'll try it on a simple program, though, since maybe that won't work either, and it will be easier to track down. – Sam Tobin-Hochstadt Jun 23 '12 at 22:39

8 Answers8

38

This post is already over a year old, but since it came out at the top of my Google search results when I had the same problem, I thought I'd answer it here. After some more searching around, I found the answer given in this related StackOverflow question very helpful. On my Ubuntu Raring system, I then ended up doing the following:

  1. Compile my C++ sources with -g (fairly obvious, you need debug symbols)

  2. Run perf as

    record -g dwarf -F 97 /path/to/my/program

    This way perf is able to handle the DWARF 2 debug format, which is the standard format gcc uses on Linux. The -F 97 parameter reduces the sampling rate to 97 Hz. The default sampling rate was apparently too large for my system and resulted in messages like this:

    Warning:
Processed 172390 events and lost 126 chunks!

Check IO/CPU overload!

    and the perf report call afterwards would fail with a segmentation fault. With the reduced sampling rate everything worked out fine.

  3. Once the perf.data file has been generated without any errors in the previous step, you can run perf report etc. I personally like the FlameGraph tools to generate SVG visualizations.

  4. Other people reported that running

    echo 0 > /proc/sys/kernel/kptr_restrict

    as root can help as well, if kernel symbols are required.

mindriot
  • 5,413
  • 1
  • 25
  • 34
  • I don't think you need an explicit `-g dwarf`. It Just Works when you build with `-g`, i.e. looking for debug info in the standard format is the default, using it if it finds any. – Peter Cordes Mar 27 '18 at 14:34
  • @PeterCordes thanks. Could be that it depends on the version and/or the distribution providing it. When I originally wrote this answer, I needed to provide the parameter explicitly, or it didn't work. – mindriot Mar 27 '18 at 14:41
  • 2
    it should be `-g --call-graph dwarf` as -g enables call-graph stack recodring but is a boolean flag only, the actual type needs to be set with the `--call-graph` mechanism. (and as using `--call-graph` implies `-g` only using `--call-graph dwarf` would be enough too) – Tom Jul 01 '21 at 09:39
6

In my case the solution was to delete the elf files which contained cached symbols from previous builds and were messing things up.

They are in ~/.debug/ folder

sotiris
  • 315
  • 6
  • 13
  • Could you please tell me who saves symbols to this folder ~/.debug? – firo Jan 09 '19 at 15:03
  • 1
    perf record command creates a folder there (mentioned [here](http://man7.org/linux/man-pages/man1/perf-config.1.html) ). But it seems that many programs write into this folder, so you'd better delete only the perf related staff. Since you normally run perf record as root , it will be in this path /root/.debug/home/username/application_name – sotiris Jan 11 '19 at 13:11
2

You can always use the '$ nm ' command.

here is some sample output:

Ethans-MacBook-Pro:~ phyrrus9$ nm a.out
0000000100000000 T __mh_execute_header
0000000100000f30 T _main
                 U _printf
0000000100000f00 T _sigint
                 U _signal
                 U dyld_stub_binder
phyrrus9
  • 1,441
  • 11
  • 26
2

I had this problem too, I couldn't see any userspace symbol, but I saw some kernel symbols. I thought this was a symbol loading issue. After tried all the possible solutions I could find, I still couldn't get it work.

Then I faintly remember that

ulimit -u unlimited

is needed. I tried and it magically worked.

I found from this wiki that this command is needed when you use too many file descriptors.

https://perf.wiki.kernel.org/index.php/Tutorial#Troubleshooting_and_Tips

my final command was

perf record -F 999 -g ./my_program

didn't need --call-graph

Bill Yan
  • 3,369
  • 4
  • 27
  • 42
1

Make sure that you compile the program using -g option along with gcc(cc) so that debugging information is produced in the operating system's native format. Try to do the following and check if there are debug symbols present in the symbol table.

$objdump -t your-elf 
$readelf -a your-elf
$nm -a your-elf
New to Rails
  • 2,824
  • 5
  • 27
  • 43
0

How about your dev host machine? Is it also running the x86_64 OS? If not, please make sure the perf is cross-compiled, because the perf depends on the objdump and other tools in toolchain.

jumper
  • 192
  • 6
0

I got the same problem with perf after overriding the name of my program via prctl(PR_SET_NAME)

As I can see your case is pretty similar:

70.06% ints.rkt [unknown]

Command you have executed (racket) is different from the one perf have seen.

Nik_
  • 146
  • 2
  • 4
0

you can check the value of kptr_restrict by cat /proc/kallsyms. If the addresses of the symbols in the result are all 0x000000, you can fix it by command echo 0 > sys/kernel/kptr_restrict . After this , you may get a wanted result of the perf report

yzark
  • 101
  • 2
  • 11