4

According to the official kernel.org documentation echo l > /proc/sysrq-trigger is supposed to give me the current call trace of all CPUs. But when I do this a couple of times and look into dmesg after that the call traces look completely similar. Why is that?

JohnnyFromBF
  • 9,873
  • 10
  • 45
  • 59
  • 1
    Possibly OT, I don't think this is a programming question. Might try Server Fault. – BadZen May 05 '15 at 17:43
  • You are doing exact same action each time: triggering sysrq. So the backtrace just showing you that fact. Try to compare backtraces for two different calls (`echo l > /proc/sysrq-trigger`) using `diff` tool, and you will see that they have different stacks, register values etc. It would be better if you show us the actual backtrace you are having. – Sam Protsenko May 05 '15 at 18:25
  • @BadZen no, it doesn't seem like offtopic to me. This question implies some knowledge about kernel internals. – Sam Protsenko May 05 '15 at 18:29
  • Which is also different than programming. – BadZen May 05 '15 at 18:30
  • Debugging is part of programming. For kernel development, using /proc/sysrq-trigger is part of debugging. Seems on-topic to me. – Gil Hamilton May 05 '15 at 22:19
  • @SamProtsenko I posted the backtrace [here](http://pastebin.com/AqGm0gnr) and the one I did 30 seconds later [here](http://pastebin.com/LLj4tVJX). The call trace is exactly the same, though the CPUs should be executing completely different functions. – JohnnyFromBF May 06 '15 at 11:22
  • @Ian yes, as I said, you are executing your sysrq command (via procfs), which leads to some code in kernel to be executed. Each time you do your sysrq command -- the same code is executed, hence you see the same backtrace every single time. Pay your attention to `write_sysrq_trigger` function in your backtrace. – Sam Protsenko May 06 '15 at 11:25
  • @Ian what exactly you are trying to achieve? You question may a subtle XY problem, and maybe what you are doing is basically the wrong solution to your original problem. – Sam Protsenko May 06 '15 at 11:29
  • Yeah I see, but I wonder how [this guy](http://askubuntu.com/a/421916/35188) got the actual backtrace doing the same thing. – JohnnyFromBF May 06 '15 at 11:29
  • 2
    @Ian That's because his CPUs were highly loaded. In your case, your CPU #0 backtrace is showing that it's executing your sysrq command (`write_sysrq_trigger()` function), and CPU #1 backtrace is showing that it's in **IDLE** state (`cpuidle_enter_state()` function). Try to load your system very intensively, and then execute your sysrq command to get new backtraces. You will see that one CPU is executing your sysrq command, and second CPU is not in IDLE anymore, but doing some actual work. – Sam Protsenko May 06 '15 at 11:50
  • That's correct indeed, I never expected that the CPU #1 would really idle all the time. Thanks for the hint! I'd like to mark your comment as the correct answer, btw. – JohnnyFromBF May 06 '15 at 13:09
  • @SamProtsenko Is there a way to see userspace functions on the stack when doing a backtrace? – JohnnyFromBF May 07 '15 at 07:19
  • 1
    @Ian I moved my comments to the actual answer. Also it has the answer to your last question in comments. – Sam Protsenko May 07 '15 at 12:02

1 Answers1

5

The same backtrace explanation

In your case, your CPU #0 backtrace is showing that it's executing your sysrq command (judging by write_sysrq_trigger() function):

delay_tsc+0x1f/0x70
arch_trigger_all_cpu_backtrace+0x10a/0x140
__handle_sysrq+0xfc/0x160
write_sysrq_trigger+0x2b/0x30
proc_reg_write+0x39/0x70
vfs_write+0xb2/0x1f0
SyS_write+0x42/0xa0
system_call_fast_compare_end+0x10/0x15

and CPU #1 backtrace is showing that it's in IDLE state (judging by cpuidle_enter_state() function):

cpuidle_enter_state+0x40/0xc0
cpu_startup_entry+0x2f8/0x400
start_secondary+0x20f/0x2d0

Try to load your system very intensively, and then execute your sysrq command to get new backtraces. You will see that one CPU is executing your sysrq command, and second CPU is not in IDLE anymore, but doing some actual work.

User-space backtrace

As for user-space functions on kernel backtrace: although system call is executing (in kernel space) on behalf of user-space process (see Comm: bash in your backtrace for CPU0), it's not possible to print user-space process backtrace using standard kernel backtrace mechanism (which implemented in dump_stack() function). The problem is that the kernel stack doesn't contain any user-space process calls (that's why you can see only kernel functions in your backtraces).

User-space process calls can be found in user-space stack for the corresponding process. For this purpose I would recommend you to use OProfile profiler. Of course, it will give you just a binary stack. In order to obtain actual function names you will need to provide symbols information to gdb.

Details:

[1] kernel stack and user-space stack

[2] how to dump kernel stack in syscall

[3] How to print the userspace stack trace in linux kernelspace

Community
  • 1
  • 1
Sam Protsenko
  • 14,045
  • 4
  • 59
  • 75