I have a windows forms (.net 3.0) project that won't run on my customer's vista computer due to a DEP error. It runs on my vista machine, and in a clean version of vista sp1 in a virtual machine. I am having trouble tracking down ways to make my program DEP, Data Execution Prevention compatible. I really can't do anything to end user machines, it just has to run. Is there any way out of this latest vista development nightmare? My program uses devexpress controls, sql express, and the .net ie web browser control. I've already jumpered out the ie control, but to no avail. I have other program that use devexpress and sql express on that same machine and they run ok. I am at a loss to debug this on the user's computer.
Asked
Active
Viewed 2.5k times
5 Answers
19
DEP runs in one of two modes:
Hardware DEP is for CPUs that can mark memory pages as non-executable. This helps to prevent certain exploits such as buffer overflows.
Software DEP is for CPUs that do not have hardware DEP support. It doesn't prevent execution of code in data pages, but instead stops SEH overwrite (another type of attack).
On Windows XP with CPUs that support it, hardware DEP is enabled by default only for certain Windows system binaries, and also for programs that choose to "opt-in".
On Vista with CPUs that support it, hardware DEP is enabled by default for nearly all processes. This can occasionally be problematic, usually for older programs and drivers, and for ISVs that haven't done any Vista testing.
So I suspect that the first step is to discover whether you're dealing with software or hardware DEP. Also, are you using C#/VB or Managed C++? And are you using any native code or components? If your application uses a native component or an ActiveX control that was built using the old ATL framework, then it's quite possible that your application will fail with hardware DEP.
Since .NET Framework 2.0 SP1, I believe that the C# compiler emits managed code which is DEP-compatible. But if your application is generating DEP exceptions, then you can try clearing the IMAGE_DLLCHARACTERISTICS_NX_COMPAT flag for your executable. To do this you use EDITBIN.EXE from the VC toolset like so:
editbin.exe /NXCOMPAT:NO <your binary>
If you're using Visual Studio, you can add a post-build step to your executable's project. You'll need to setup the environment so that EDITBIN's dependencies can be resolved. When I'm using native code as part of my app, the post-build step looks like this:
call $(DevEnvDir)..\tools\vsvars32.bat
editbin.exe /NXCOMPAT:NO $(TargetPath)
General Grievance
- 4,555
- 31
- 31
- 45
HTTP 410
- 17,300
- 12
- 76
- 127
5
Older versions of ATL are not DEP aware, so if you use any ActiveX controls built using ATL and were built on that version of ATL (version 7.1 and below, I think), you'll get DEP errors.
As a last resort, you can disable DEP for the process by calling an API function: SetProcessDEPPolicy.
More information on SetProcessDEPPolicy.
oguz ismail
- 1
- 16
- 47
- 69
mackenir
- 10,801
- 16
- 68
- 100
5
The compilers that shipped with .NET 2.0 SP1 turn on the NXCOMPAT flag in the executable file header. You can turn that flag off in a Post Build step by running EditBin.exe with the /NXCOMPAT:NO option.
Hans Passant
- 922,412
- 146
- 1,693
- 2,536
3
FWIW, it's worth explicitly mentioning that in many cases, applications aren't "incompatible with DEP" but rather were about to crash anyway and DEP "dove in to save the day." Very often, once you disable DEP, you'll find that you're hitting an "ordinary" AV.
If your project is written solely in .NET 3.0, this is almost certainly the case, because .NET doesn't do any of the "crazy" things that trigger DEP (e.g. function thunking, etc).
To debug, install a debugger or enable Watson to generate a .DMP file, then take that .DMP file to the developer's machine and figure out what went wrong.
EricLaw
- 56,563
- 7
- 151
- 196
-
The problem is with an older activex control that I have to use with my program. It controls a meter, and the main purpose of the app is to run the meter. I have been unable to get the vendor to rebuild the app with newer atl libraries. The app also has to run on all end user machines, and I am hitting dep bugs on some of them. I have no control or management of the user machines directly. – P a u l Jul 11 '09 at 03:48
-
Ah. Well, one choice would be to use SetProcessDEPPolicy to opt-in to DEP rather than using the NXCOMPAT linker option. When you use SetProcessDEPPolicy to turn on DEP, it leaves ATL Thunk Emulation enabled, unlike the linker option. This is how IE8 turns on DEP in order to remain compatible with ancient ATL controls. – EricLaw Jul 11 '09 at 14:48
0
Start by trying to figure out where and how your program is failing. Can you replicate the issue on your system? With enabling DEP for the application on your system? When you can replicate the issue and get the error (access violation), you can look to fixing your program.
See the MSDN article for information on DEP.
Craig
- 2,248
- 1
- 19
- 23
Kris Kumler
- 6,307
- 3
- 24
- 27
-
It's a completely opaque situation. I get no details from vista, I have no debugger on the system. The program runs fine on my dev vista machine and in a clean vista virtual machine. – P a u l Dec 08 '08 at 21:56