0

I have a huge application (made in PowerBuilder) that crashes every once in a while so it is hard to reproduce this error. We have it set up so that when a crash like this occurs, we recieve a .dmp file.

I used WinDbg to analyze my .dmp file with the command !analyze -v. From this I can deduct that the error that occured was an Access Violation C0000005. Based on the [0] and [1] parameters, it attempted to dereference a null pointer.

WinDbg also showed me STACK_TEXT consisting of around 30 lines, but I am not sure how to read it. From what I have seen I need to use some sort of symbols.

First line of my STACK_TEXT is this:

00000000`00efca10 00000000`75d7fa46     : 00000000`10df1ae0 00000000`0dd62828 00000000`04970000 00000000`10e00388 : pbvm!ob_get_runtime_class+0xad

From this, my goal is to analyze this file to figure out where exactly in the program this error happened or which function it was in. Is this something I will be able to find after further analyzing the stack trace?

How can I pinpoint where in the program a crash happened using .dmp and WinDbg so I can fix my code?

rogic
  • 13
  • 4
  • Dump analysis requires a lot of experience and usually is an expensive consulting service provided by a few vendors (including Microsoft), so don't assume SO is the place (especially for complicated access violation category). – Lex Li Nov 10 '22 at 21:55
  • @LexLi as someone with no experience with this, what do you think I should do next? – rogic Nov 10 '22 at 22:16
  • Purchase the consulting services if I didn't make it clear in previous comment. – Lex Li Nov 10 '22 at 22:49
  • @LexLi yes I got that I could purchase consulting services that's probably the easiest way to go about this but is there anything I could do without purchasing them. – rogic Nov 11 '22 at 16:40
  • Basically the approach is exactly as you did: capture a crash dump file and analyze it. You need symbols (PDBs) and [set them up correctly](https://stackoverflow.com/questions/30019889/how-to-set-up-symbols-in-windbg), then look at the faulting call stack and correlate that to your program. Crashes are relatively easy to debug, because you have the call stack – Thomas Weller Nov 13 '22 at 10:33
  • `pbvm!ob_get_runtime_class+0xad` is a symbol and a quite good one. If `pbvm` is your DLL, you're quite close already. If it isn't, look at the first line that belongs to your code. – Thomas Weller Nov 13 '22 at 10:34
  • @ThomasWeller thank you for the thoughtful response! I am currently attempting setting up symbols from the link you sent. Yes the pbvm is my DLL. Do you know what the "ob_get_runtime_class" means? My idea was that it could be the function name inside the DLL. And then I would assume setting up symbols would convert this "00000000`00efca10 00000000`75d7fa46" into something readable? – rogic Nov 21 '22 at 17:08

1 Answers1

0

If you analyze a crash dump with !analyze -v, the lines after STACK TEXT is the stack trace. The output is equivalent to kb, given you set the correct thread and context.

The output of kb is

  1. Child EBP
  2. Return address
  3. First 4 values on the stack
  4. Symbol

The backticks ` tell you that you are running in 64 bit and they split the 64 bit values in the middle.

On 32 bit, the first 4 parameters on the stack were often equivalent to the first 4 parameters to the function, depending on the calling convention.

On 64 bit, the stack is not so relevant any more, because with the 64 bit calling convention, parameters are passed via registers. Therefore you can probably ignore those values.

The interesting part is the symbol like pbvm!ob_get_runtime_class+0xad. In front of ! is the module name, typically a DLL or EXE name. Look for something that you built. After the ! and before the + is a method name. After the + is the offset in bytes from the beginning of the function.

As long as you don't have functions with thousands of lines of code, that number should be small, like < 0x200. If the number is larger than that, it typically means that you don't have correct symbols. In that case, the method name is no longer reliable, since it's probably just the last known (the last exported) method name and a faaaar way from there, so don't trust it.

In case of pbvm!ob_get_runtime_class+0xad, pbvm is the DLL name, ob_get_runtime_class is the method name and +0xad is the offset within the method where the instruction pointer is.

To me (not knowing anything about PowerBuilder) PBVM sounds like the PowerBuilder DLL implementation for Virtual Memory. So that's not your code, it's the code compiled by Sybase. You'd need to look further down the call stack to find the culprit code in your DLL.

After reading Wikipedia, it seems that PowerBuilder does not necessarily compile to native code, but to intermediate P-Code instead. In this case you're probably out of luck, since your code is never really on the call stack and you need a special debugger or a WinDbg extension (which might not exist, like for Java). Run it with the -pbdebug command line switch or compile it to native code and let it crash again.

Thomas Weller
  • 55,411
  • 20
  • 125
  • 222
  • And for the SAP tool chain, the rule is to spend $3000 on support and get some professional help. – Thomas Weller Nov 21 '22 at 18:29
  • That makes so much more sense thank you. Analyzing my STACK TEXT even more - it contains 40 lines of it. The first 35 lines have `PBVM!` in them. Then it is followed by two lines, `ns2+0x2338` and `ns2+0xc1d8` (**NS2** is the name of my PowerBuilder app), and the last two are `kernel32!BaseThreadInitThunk+0x14` and `ntdll!RtlUserThreadStart+0x21`. So because I have these two entries that have my apps name on it, could I assume that's where to app crashed? This is a very big app so it contains a lot of code. Since you mentioned `<0x200` this means I can't use 0x2338 and 0xc1d8? – rogic Nov 21 '22 at 19:27
  • Also what is the easiest type of error I could simulate that would result in a .dmp file. My current app is way too big and could take weeks to crash again. I'm thinking of just writing a "Hello World" type problem, create any error, and then analyze that file inside WinDbg. Then I could compare the two and see the difference. Maybe it would give me new insight to the `pbvm!` and how to use the offset in bytes to find where in the code the crash happened. – rogic Nov 21 '22 at 19:51
  • @rogic: it's a good idea to start debugging with something simple. Try dividing a number by 0, not hardcoded, though, because that might be detected by the compiler. Can you have 2 numbers as user input and divide them? – Thomas Weller Nov 21 '22 at 20:00
  • Check how you can generate PDBs, i.e. files that will contain symbols for the `ns2` DLL. As you can see, there is not even a method name printed. – Thomas Weller Nov 21 '22 at 20:01
  • Yep I can! I'll start with that then. By the way, I was emailed this .dmp file so I am not sure in which directory they are usually get created in. Are they in the same directory as the app/code or does Windows place them somewhere completely different? – rogic Nov 21 '22 at 20:06
  • @rogic: it seems you need Search Everything. You'll love it. https://www.voidtools.com/downloads/ – Thomas Weller Nov 21 '22 at 20:07
  • Also check out [WER LocalDumps as described here](https://stackoverflow.com/a/24874028/480982) – Thomas Weller Nov 21 '22 at 20:10
  • How do you know which are the right symbols to use? From the link you sent me, I can see that I am able to setup symbols provided by Microsoft and then you also mention I can setup local symbols. How will I know if Microsofts symbols are right, and if they aren't how do I obtain the local ones. I didn't make any changes to symbols so I should be using the default ones when first installing WinDbg. I am analyzing the file with the "dividing by 0" error but the offsets are way too big. 0x120b2, 0x121f8, 0x43922, etc. I wonder if the `PBVM` is what could be messing these offsets up. – rogic Nov 22 '22 at 18:18
  • Microsoft symbols will be downloaded from a webserver. I don't know where the PBVM symbols came from, but they look good. WinDbg (by default) will only load symbols that match the DLL (by checksum and timestamp), so there's hardly a chance that wrong symbols are loaded. Use `.sympath+` to add local paths. See also: [How to set up symbols](https://stackoverflow.com/a/30019890/480982) – Thomas Weller Nov 22 '22 at 18:22
  • I did a comparison, C++ vs Powerbuilder crash. When I get a crash from C++, analyzing the dump file inside WinDbg gives me all the details, like the name of the function. It even gives me the faulting source code and highlights exactly where the crash happens. Powerbuilder doesn't show any of that. I'm wonder if the PBVM is really what is causing a problem. Maybe I am just out of luck when dealing with PowerBuilder crash dumps. – rogic Nov 28 '22 at 13:05
  • Did you build to P-Code or native code? Did you find that setting? – Thomas Weller Nov 28 '22 at 13:57
  • It was P-Code, I'm trying to figure out how to do native code - or if it's even possible. Posted a question to the PowerBuilder community to see what they say. – rogic Nov 29 '22 at 19:04
  • Compiled it in Machine Code, however I still need the same PB Runtime files that a P-Code executable does. So it still uses PBVM and WinDbg shows about the same output as the P-Code does unfortunately. I think I might be out of luck with this one.. – rogic Nov 29 '22 at 20:38
  • @rogic: yes, some things are better not debugged in WinDbg. Like Java. – Thomas Weller Nov 29 '22 at 20:39
  • `0019d4e8 02513600 04355fb4 00000000 00000005 pbvm!ot_cause_gpf+0x12 \n 0019d59c 794a6636 04355fb4 00000000 04207a1c crashdumptest+0x3600 \n 00000000 00000000 00000000 00000000 00000000 pbvm!ob_get_pcode_stack_effect+0x246` This is what the stack text shows me. I would assume the middle line is where it crashed, but unfortunately it doesn't show anything useful except the executable name. I wish I could see details like code and function name like how C++ does it. – rogic Nov 29 '22 at 20:43
  • What's used for Java? – rogic Nov 29 '22 at 20:43
  • It seems you don't have symbols for crashdumptest at all. It just has an offset but no method name. For java: IntelliJ, Eclipse or JDB. – Thomas Weller Nov 29 '22 at 20:45
  • Ok I'll try those, maybe one of them works. I'll just ask you one last questions, since you have helped me a lot so far and I've been asking a lot of questions. How or where could I acquire the PBVM symbols and what kind of file extension am I looking for? – rogic Nov 29 '22 at 21:07
  • You have PBVM symbols. See `lmf m pbvm` You don't have `crashtestdump` symbols. You are looking for a file with PDB extension. – Thomas Weller Nov 29 '22 at 21:18
  • Oh I see what you mean. I'll try to find symbols for crashtestdump. Thanks for all you help sir! – rogic Nov 29 '22 at 22:20