3

I am using VS 2010 on a win 7 64 bit system with 8 GB of memory. My application is 32 bit. While in the VS 2010 .Net IDE, the app shows up in the Windows task manager as "MyApp.vshost.exe *32" while the VS IDE itself shows up as "devenv.exe *32".

I checked and it appears that the VS 2010 IDE file (devenv.exe) is complied with the /LargeAddressAware flag.

However, when debugging large models, the IDE fails with an Out of memory exception. In the Windows Task manager, the "MyApp.vshost.exe *32" process indicates about 1400 MB of memory usage (while the "devenv.exe *32" process is well under 500 MB). Is it possible to set the "MyApp.vshost.exe *32" process to be /LargeAddressAware in order to avoid this out of memory situation? If so, how can this be done in the IDE. While setting the final application binary to be /LargeAddressAware would work, I still need to be able to debug the app in the IDE with these type of large models. I should also note that my app has a deep object hierarchy with many collections that together required a lot of memory. However, my issue is not related to trying to create say 1 large array that requires greater then 2 GB of memory etc.

I should note that I am able to run the same app in the VB6 IDE and not get an out of memory situation as long as the VB6 IDE is made /LargeAddressAware. In the case of VB6, the IDE and the app being debugged are part of the same process (and not split into 2 as is the case with VS 2010.) The VB6 process can be larger then 3 GB without running into out of memory issues.

Ultimately, my objective is to have my app run completely in 64 bit to access more memory. I am hoping that in such cases, the IDE will allow the debugging process to exceed 2 GB without crashing (and certainly more then 1.4 GB as is the current case). However, for now, while 95% of my app is 64 bit, I am calling a legacy COM 32 bit DLL and as such, my entire app is forced to still run in 32 bit mode until I replace that DLL.

David Schenk
  • 128
  • 10
user561732
  • 83
  • 1
  • 4
  • That is the plan. The dll in question is MS DirectX 8. Will need to update to MS DirectX 10 or 11. Unfortunately, this is not a trivial process! – user561732 Jan 03 '11 at 23:12

2 Answers2

3

the IDE fails with an Out of memory exception

No, your program fails with that exception, not the IDE. You'd have to run editbin.exe in a post build event to set the flag:

set path=%path%;$(devenvdir);$(devenvdir)\..\..\vc\bin
editbin /largeaddressaware $(targetpath)

That will not work on the vshost.exe version, you'll have to turn off the hosting process. Project + Properties, Debug tab.

Hans Passant
  • 922,412
  • 146
  • 1,693
  • 2,536
  • Awesome, I turned off the optional hosting process as you suggested (and Paolo in the first answer) and then, manually used Laatido app to set myApp.exe to /LargeAddressAware. I was then able to run one of my large models in the IDE! I will later try to use the Post build approach to better automate the steps. – user561732 Jan 03 '11 at 22:25
  • What the heck is a "Laatido app"? You paid somebody to turn that bit on? – Hans Passant Jan 03 '11 at 22:28
  • No, I could not find EditBin on my last computer and at that time, downloaded a free applet that has a simple form that lets you pick a exe file, check if it already has the /LargeAddressAware flag set and if not, sets the flag for you. – user561732 Jan 03 '11 at 22:33
  • That's a liability of using the free Express edition. If you don't have the money then you'd better have the time to find alternatives. Spending it on RAM then? – Hans Passant Jan 03 '11 at 22:35
  • No, I am using the VS 2010 Pro edition. I was not familiar with EditBin and had doanloaded laatido. As indicated above, I will try using editBin in a postBuild to automate the steps. – user561732 Jan 03 '11 at 22:38
  • Then my postbuild event snippet should work just fine. Not sure why you didn't try it. You have to give a diagnostic if it doesn't work for you, I can't guess error messages without a web cam. – Hans Passant Jan 03 '11 at 22:59
  • I have not tried the PostBuild approach yet. Never have. Where is the PostBuild accessed in VS 2010? – user561732 Jan 03 '11 at 23:08
  • Project + Properties, Build Events tab. Click around some more to see what's available. Press F1 if you don't know what it means. – Hans Passant Jan 03 '11 at 23:12
2

Some options I can think of:

  1. Disable the VSHost process - Do you actually require the extra debugging features from the VS hosting process? If not just to untick the "Enable Visual Studio Hosting Process" option.

  2. Externalise the problematic DLL - Wrap the COM DLL in a simple 32-bit process and compile the rest as 64-bit using an appropriate variety of IPC to connect the two.

  3. Force the flag in the VSHost process - Use a post-build command to forcibly set the flag in the .vshost.exe (no idea if that would work though!)

Community
  • 1
  • 1
Paolo
  • 22,188
  • 6
  • 42
  • 49