The trick is not to use VDMEnumProcessWOW (which gives the VDMs), but to use VDMEnumTasksWOW. The enumerator function that you pass to this function will be called for each 16 bit task in the specified VDM.
I haven't checked it myself, but according to the documentation, this library of CodeProject does exactly that, if you pass in the PROC16 enum value. It's C++, if you need help compiling that code and calling it from C#, let me know and I'll give you an example.
A program that uses this technique is Process Master, it comes with full source. I suggest you run it to find out whether it gives the info you need, and if so, you can apply this method to your own application (it doesn't run on Windows Vista or 7, it uses old VB5 code, apparently it's not compatible. It should run on XP).
If things with these functions do not go as planned, you may be on Vista and may need the hotfix described in this StackOverflow question, which points to downloading a hotfix, which is in turn described here:
"An application that uses the
VDMEnumProcessWOW function to
enumerate virtual DOS machines returns
no output or incorrect output on a
computer that is running a 32-bit
version of Windows Vista"
Update: while this seems promising, I applied the patch, ran several versions of the code, including Microsoft's, and while they all work on XP, they fail silently (no error, or wrong return value) on Vista.
The "kinda" working code
Update: I experimented with (amongst others) with the following code, which compiles fine in C# (and can be written simpler, but I didn't want to run a marshal-mistake risk). When you add these functions, you can call Enum16BitProcesses, which will write the filenames of the EXE files of the 16 bit processes to the Console.
I can't run it on Vista 32 bit. But perhaps others can try and compile it, or find the error in the code. It would be nice to know whether it works on other systems:
public class YourEnumerateClass
{
public static void Enum16BitProcesses()
{
// create a delegate for the callback function
ProcessTasksExDelegate procTasksDlgt =
new ProcessTasksExDelegate(YourEnumerateClass.ProcessTasksEx);
// this part is the easy way of getting NTVDM procs
foreach (var ntvdm in Process.GetProcessesByName("ntvdm"))
{
Console.WriteLine("ntvdm id = {0}", ntvdm.Id);
int apiRet = VDMEnumTaskWOWEx(ntvdm.Id, procTasksDlgt, IntPtr.Zero);
Console.WriteLine("EnumTaskWOW returns {0}", apiRet);
}
}
// declaration of API function callback
public delegate bool ProcessTasksExDelegate(
int ThreadId,
IntPtr hMod16,
IntPtr hTask16,
IntPtr ptrModName,
IntPtr ptrFileName,
IntPtr UserDefined
);
// the actual function that fails on Vista so far
[DllImport("VdmDbg.dll", SetLastError = false, CharSet = CharSet.Auto)]
public static extern int VDMEnumTaskWOWEx(
int processId,
ProcessTasksExDelegate TaskEnumProc,
IntPtr lparam);
// the actual callback function, on Vista never gets called
public static bool ProcessTasksEx(
int ThreadId,
IntPtr hMod16,
IntPtr hTask16,
IntPtr ptrModName,
IntPtr ptrFileName,
IntPtr UserDefined
)
{
// using PtrToStringAnsi, based on Matt's comment, if it fails, try PtrToStringAuto
string filename = Marshal.PtrToStringAnsi(ptrFileName);
Console.WriteLine("Filename of WOW16 process: {0}", filename);
return false; // false continues enumeration
}
}
Update: Intriguing read by the renown Matt Pietrek. Mind the sentence, somewhere near the end:
"For starters, MS-DOS-based programs
seem to always run in separate NTVDM
sessions. I was never able to get an
MS-DOS-based program to run in the
same session as a 16-bit Windows-based
program. Nor was I able to get two
independently started MS-DOS-based
programs to run in the same NTVDM
session. In fact, NTVDM sessions
running MS-DOS programs don't show up
in VDMEnumProcessWOW enumerations."
Seems that, to find out what processes are loaded, you'll need to write a hook into NTVDM or write a listener that monitors access to the file. When the application that tries to read a certain DOS file is NTVDM.exe, it's bingo. You may want to write a DLL that's only attached to NTVDM.exe, but now we're getting a bit ahead of ourselves. Long story short: this little ride into NTVDM has shown "possibilities" that appeared real hoaxes in the end.
There's one other way, but time is too short to create an example. You can poke around in the DOS memory segments and the EXE is usually loaded at the same segment. But I'm unsure if that eventually will lead to the same result and whether it's worth the effort.
