2

Solved: For when simple profiling isn't effective enough, I have written a tool to show me where performance hits occur. Basic information about how the tool works is in the accepted answer below. The source can be found here: http://pastebin.com/ETiW8hE8 (be sure to turn debugging symbols on in the program you're testing)

I've built a game engine in C++ and I have noticed in one particular area of a level that there is a brief performance hit. The game will stop completely for about half a second, and then continue on merrily. I've tried to profile this, but it's difficult isolate the condition since I also have to load the map and perform the in-game task which causes the performance hit. I can make a map load automatically and skip showing menus, etc, and comparing those profile results against a set of similar control data (all the same steps but without actually initiating the performance hit), but it doesn't show anything obvious.

I'm using gmon to profile.

This is a large application with many, many classes and functions. The performance hit only happens once, so there's no way to just trigger the problem many times during one execution to saturate my profiling results in order to make the offending functions more obvious in the profiling results.

What else can I do?

Brandon
  • 483
  • 3
  • 12

2 Answers2

2

What I would do is try to grab a stack sample in that half second when it's frozen. This would require an alarm clock timer set to go off some small time in the future, like 100ms. Then in some loop, like the frame display loop, that normally takes less than 100ms to repeat, keep resetting the timer. That way, it will act as a watchdog that barks if you don't keep petting it.

Then, stick a breakpoint in the timer interrupt handler. When it gets there, you know you're in the bad slice of time. Then just display the call stack, and it should show you what the problem is.

You might have to repeat the process a few times.

Community
  • 1
  • 1
Mike Dunlavey
  • 40,059
  • 14
  • 91
  • 135
  • Well, I have a program now that expects input every X (mili)seconds and growls at me if I don't give it some. So, I've inadvertently created another (bad) game in the process. :) Will update later. – Brandon Apr 23 '14 at 23:55
  • 1
    Behold! My new game, AngryPet: http://pastebin.com/S0pmWjw1 ... okay, back to work now. The idea is going to be for my game to pipe output to this one, and when there's a big delay it'll gimme a stack trace with pstack, as mentioned by Soren. – Brandon Apr 24 '14 at 00:14
  • Got it! I've got a stack trace now during the performance hit. I still have to study exactly what's going on but the tool seems to work. Also, +1 to debugging, multi-threading, and inter-process communication skills for me! Here's the code for the tool: http://pastebin.com/ETiW8hE8 – Brandon Apr 24 '14 at 01:33
  • 1
    Update: The tool led to the discovery of what was going on: I was packing entity model VBOs when they happened to come on screen, not during level load. Oops! – Brandon Apr 24 '14 at 02:31
2

You are not saying anything about whether your application is threaded, but I will assume that it is not.

As per suggestion from mike, get insights by getting a stack trace at and see where it is freezing, you can do that with a bit of luck using pstack, so

   while usleep 100000; do 
      pstack processid
   done >/tmp/stack.log

Should give you some output to go on -- my guess is that you are calling a blocking IO operation, like reading some assets from disk.

Soren
  • 14,402
  • 4
  • 41
  • 67
  • I use SDL which I believe is multithreaded, but my application itself is not. – Brandon Apr 23 '14 at 21:28
  • 1
    It looks like `pstack` and `gstack` may be the same -- the man page says `If the process is part of a thread group, then gstack will print out a stack trace for each of the threads in the group` – Soren Apr 23 '14 at 21:36