I am trying to attach a debugger to a 3rd party program to do some reverse engineering. This program is saving zip files that are not normal zip files, it can open them fine but I get errors when I try to use software like 7zip or CSharpZipLib. I want to attach a debugger see how the program is doing it's unzipping. Also the "malformations" are not the same in every zip file. Some unzip fine with no modification, some have malformed headers like in my linked question, some appear to be fine, but CSharpZipLib fails to unzip it (but window's built in unzip works fine).
I know it is calling something in java when it is unzipping as using ProcessExplorer I can see very many calls to rt.jar from the exe while it is writing out a unzipped version to the temp folder (I need to do it myself as I need to unzip 105,987 zip files and it would take a unreasonable amount of time to do it via the UI of the program and grab the unzipped file from the temp folder).
From this SO question I found how to debug the program when it does not have the debugger arguments started with the JVM. I download VisualVM and I can attach and profile the program, but I can not find any way to get a function call trace from inside VisualVM.
I also tried the jsadebugd solution, but I get the following error when I try to start the program.
C:\>"c:\Program Files\Java\jdk1.6.0_34\bin\jsadebugd.exe" 8824
Attaching to process ID 8824 and starting RMI services, please wait...
Error attaching to process or starting server: sun.jvm.hotspot.debugger.DebuggerException: Windbg Error: ReadVirtual failed!
at sun.jvm.hotspot.debugger.windbg.WindbgDebuggerLocal.readBytesFromProcess0(Native Method)
at sun.jvm.hotspot.debugger.windbg.WindbgDebuggerLocal.readBytesFromProcess(WindbgDebuggerLocal.java:485)
at sun.jvm.hotspot.debugger.DebuggerBase.readBytes(DebuggerBase.java:219)
at sun.jvm.hotspot.debugger.DebuggerBase.readCInteger(DebuggerBase.java:375)
at sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:454)
at sun.jvm.hotspot.debugger.windbg.WindbgDebuggerLocal.readAddress(WindbgDebuggerLocal.java:316)
at sun.jvm.hotspot.debugger.windbg.WindbgAddress.getAddressAt(WindbgAddress.java:72)
at sun.jvm.hotspot.HotSpotTypeDataBase.readVMTypes(HotSpotTypeDataBase.java:126)
at sun.jvm.hotspot.HotSpotTypeDataBase.<init>(HotSpotTypeDataBase.java:85)
at sun.jvm.hotspot.HotSpotAgent.setupVM(HotSpotAgent.java:388)
at sun.jvm.hotspot.HotSpotAgent.go(HotSpotAgent.java:315)
at sun.jvm.hotspot.HotSpotAgent.startServer(HotSpotAgent.java:221)
at sun.jvm.hotspot.DebugServer.run(DebugServer.java:107)
at sun.jvm.hotspot.DebugServer.main(DebugServer.java:46)
at sun.jvm.hotspot.jdi.SADebugServer.main(SADebugServer.java:63)
I do have the "Debugging Tools For Windows" installed under the path C:\Program Files\Debugging Tools for Windows (x64). I also have windbg.exe located under C:\Program Files (x86)\Windows Kits\8.0\Debuggers\x64 and C:\Program Files (x86)\Windows Kits\8.0\Debuggers\x86. Could the multiple copies be my problem, and if so, how do I fix it?
What do I need to do to watch what this exe is doing so I can mimic it's zip decompression routine?
