6

I find myself in the difficult situation of having to debug a Qt application without almost any debugging tool: the application seems to start using more and more CPU as it is running the same action again and again; after many hours CPU is completely saturated.

The application runs on a ARM Linux embedded device where gdb seems not to work, maybe hard-to-discover problems with the provided toolchain. strace seems only to report timer activities (this is an OpenGL application so this is expected). ltrace is not available and compiling it resulted in a difficult task, maybe useless. I didn't write the application but the source code is available.

Is there anything else I can do to discover what the application is busy doing when consuming that much resources? Any way I have to trace all the method calls the application does? Is there any other technique I can use to try to guess the problem or where to focus my attention?

EDIT: This is one of the problems with gdb: Only question marks in backtrace reported by gdb on ARM. Even writing a ten lines application simulating a segfault results in this.

Community
  • 1
  • 1
Luca Carlon
  • 9,546
  • 13
  • 59
  • 91
  • Have you tried Remote Debugging ? – manasij7479 Mar 21 '12 at 07:28
  • Remote debugging with gdb? Tried for hours without success. valgrind has problems as well. No one was able to make these tools working om this platform. Also consider that system libs are all stripped. – Luca Carlon Mar 21 '12 at 07:38
  • 1
    Can you output texts through external ports, e.g. serial port or over network? Then you can add logging over that. Or just write logging to a file. – Some programmer dude Mar 21 '12 at 07:45
  • Yes, I can place logs everywhere I want. But consider that this is a large code, with thousands of lines using also some external libraries where the bug might be hiding. Is there any way to place logs automatically? – Luca Carlon Mar 21 '12 at 07:56
  • You mentionned that the source was available, I would seriously consider recompiling on a well-supported platform (any *nix system really) and see if the bug happens there. Chances are the bug is applicative and not platform dependent. As a bonus, you'll get debug symbols. – Matthieu M. Mar 21 '12 at 08:04
  • Unfortunately, the application uses libraries which are only available for that platform and hardware (hardware accelerated), and for which source code is not available. Testing those libraries suggests those are not causing the issue. If I was able to compile for another platform I could simply use x86 with gdb. Unfortunately in this case it can't be done. Anyway, I can compile the applications with symbols. – Luca Carlon Mar 21 '12 at 08:15
  • http://secretgeek.net/image/real-programmers-code-in-binary_ftfy.png – valdo Mar 21 '12 at 09:19

5 Answers5

4

Can you enable core dumps on the machine? Then when it is playing up, you can send it a SIGABRT and copy the core dump to your dev machine, and examine it with a cross-debugger, with source and unstripped executable available.

It's also important to learn the bitter lesson for next time, don't use such a badly supported toolchain.

If it's an option, you could try another toolchain with at least gdbserver if not gdb support. I have been quite happy with the CodeSourcery ARM Lite toolchain.

EDIT: gdb for your situation comes in two flavours:

  • a cross-gdb that runs on your development host
  • a native gdb that runs on your target

gdbserver allows you to run your cross-gdb on your development host and connect to the target to remotely debug something running on it. So a core dump or gdbserver are two ways to use a cross-gdb to inspect something on the target, but gdbserver alone won't help you much.

If your cross-compiler is something like arm-none-linux-gnueabi-gcc, see if you have an arm-none-linux-gnueabi-gdb available on your dev host.

blueshift
  • 6,742
  • 2
  • 39
  • 63
  • This is very interesting! But I see I need gdb to read the dump. I don't think I have gdb available for the platform, only gdbserver for remote debugging, which also seems not to be working correctly. Can I try this with that tool? – Luca Carlon Mar 21 '12 at 08:18
  • Is this some very unusual ARM device? If not, you should be able to easily build gdb for it from sources. – dbrank0 Mar 21 '12 at 08:58
  • This is a good advice. Thanks! This is a device with an embedded Linux built from scratch. – Luca Carlon Mar 21 '12 at 11:38
3

You can try to place some debugging code in your application.

Choose some signal, like SIGINT. Add signal handler for this signal. In this handler print the stack trace or at least the instruction pointer value. Then start application and send SIGINT several times to see what your application is doing.

Evgeny Kluev
  • 24,287
  • 7
  • 55
  • 98
  • Yes, I've already tried this. But like gdb it seems that I cannot have the name of the functions in the backtrace, which makes the backtrace useless. backtrace_symbols seems to be unable to match the pointer to the string, even when symbols are in my binary. – Luca Carlon Mar 21 '12 at 08:54
  • You can always print the instruction pointer. And then find it in the map file. – Evgeny Kluev Mar 21 '12 at 09:05
  • I tried this on a sample code and seems to work when exporting symbols with rdynamic! Problem now is that for some unknown reason in the application I'm analyzing I can't catch SIGINT. – Luca Carlon Mar 21 '12 at 11:36
  • 1
    May be you cannot catch it because it is already catched somewhere in the application... Try some other signal then. – Evgeny Kluev Mar 21 '12 at 12:45
0

Try logging the execution time of different functions. First log the execution time of the most likely candidates and if you have eliminated them continue with the other less likely functions in your program.

The simplest way to log a message is to use std::cout (or printf) and redirect the out to a file so you can see the log in a latter time.

Svetlin Mladenov
  • 4,307
  • 24
  • 31
0

You could try running the ARM version of the Zoom profiler - this should tell you where your code is spending most of its time - you can download it for free on a 30 day evaluation license.

Paul R
  • 208,748
  • 37
  • 389
  • 560
0

Assuming gcc has something like MSVC file and line macros, which expand to the current file and the current line, you can make your own pseudo-profiling function. Put this in a header:

void MyPseudoProfileLog(const char* file, int line, int* count);
#define MY_PSEUDO_PROFILE_LOG(file, line) { static int count = 0; MyPseudoProfileLog(file, line, &count); }

This goes in a cpp file (if you put it in the header you will get multiple copies of the static variable, one for each cpp file you include the header in):

void MyPseudoProfileLog(const char* file, int line, int* count)
{
    static FILE* f = NULL;
    const int max_count = 1000;
    if (++(*count) == max_count)
    {
        *count = 0;
        if (!f)
            f = fopen("/tmp/my_pseudo_profile_log.log");
        fprintf(f, "file=\"%s\", line=%d was passed %d times\n", file, line, max_count);
        fflush(f);
    }
}

Then you can paste

MY_PSEUDO_PROFILE_LOG(__FILE__, __LINE__);

to various places in your code to see how often they get called. Keep in mind that this is not thread-safe so use only in the main thread.

sashoalm
  • 75,001
  • 122
  • 434
  • 781
  • Placing log lines is what I'm doing yes, problem is there are hundreds of method calls in various libraries. It might take weeks to find those loading the CPU. – Luca Carlon Mar 21 '12 at 10:12