0

One big promise of Delphi 10 is the reduction of out of memory errors. This works because the IDE can now allocate addresses larger than 2GB and defines a special flag in the PE-Header for bds32.exe:

PEView for bds.exe

We have an autmated build environment with MSBUILD, which is calling dcc32.exe directly. Sadly, dcc32.exe does not include this flag and is unable to compile some projects because there is not enough memory available.

enter image description here

ventiseis
  • 3,029
  • 11
  • 32
  • 49
  • Try and and see. Enable top down memory allocation in the registry, reboot, and stress test it. Expect to have to disable any anti-malware programs when you do that. – David Heffernan Jan 22 '16 at 09:23
  • If I understand the issue correctly, this is a bug. I'll create a bug report if I'm sure I undertand the issue correctly... – ventiseis Jan 22 '16 at 09:23
  • It's not necessarily a bug. Perhaps the program cannot cope with addresses above 2GB. Older Borland executables based on the old Borland MM could not. And dcc32 is a somewhat ancient program. Maybe it is still afflicted. – David Heffernan Jan 22 '16 at 09:24
  • 2
    It's only a bug if it's not working as designed. It's designed not to support large addresses, and it does not support large addresses. I've never had the command line compiler need enough memory for this to be an issue, and I use it regularly with MSBuild. And despite what David has said, DCC32 is not a *somewhat ancient program*; it's currently maintained and has had substantial changes recently (in order to support new language features). And the big promise of Delphi 10 was more memory for the IDE itself, and it has that; nothing was every said about more memory for DCC32. – Ken White Jan 22 '16 at 13:55
  • 1
    @KenWhite Ancient programs get maintained. It certainly is a somewhat ancient program. It was certainly developed a very long time before the large address aware concept was introduced. And it's therefore plausible that Emba have not tackled the changes needed to make it support >2GB addresses. So, that's why I'd test with top down memory allocation. – David Heffernan Jan 22 '16 at 14:16
  • @KenWhite For me as an user, this is a bug. Why can I compile a project in the IDE and not in the command line? By the way, I patched dcc23.exe to have the large address aware flag and it seems to produce a correct compilation result without the out of memory error. To be clear: I won't use this in production to compile release versions. – ventiseis Jan 22 '16 at 14:19
  • 1
    @ventiseis Did you enable top down memory allocation? http://stackoverflow.com/a/545097/505088 You really ought to do that, reboot, and then retest your modified dcc32. If it works under those conditions then I think it's safe to use for production. But as I said, you might find that tools like anti-malware have problems running under top down allocation. – David Heffernan Jan 22 '16 at 14:19
  • No, I didn't, but I will if you recommend this! – ventiseis Jan 22 '16 at 14:24
  • Ventiseis: As I said, a bug is when something does not work as designed. It may be a *feature you'd like to have*, but it's not a *bug*. As for why it wasn't changed in the command line compiler, that's a question you should be asking in the EMBT forums instead of here. – Ken White Jan 22 '16 at 14:24
  • 1
    You really need to do this to make sure you flush out any > 2GB bugs. Doesn't guarantee that you get them all, but when I enabled large address aware back in the D6 days, top down testing like this was exceptionally effective. – David Heffernan Jan 22 '16 at 14:25
  • 2
    @KenWhite I won't start a discussion about that with you. If I **buy** an IDE I expect the compiler at least to compile reasonable inputs. And I will report this at the quality portal because for me this is an issue. – ventiseis Jan 22 '16 at 14:30
  • @DavidHeffernan Thank you for that hint about top-down-allocation. I'll enable that. We use the automated build for an automated testing environment which requires linking with debugging information. For production releasses we don't link with debugging information and the project can be compiled fine with msbuild. – ventiseis Jan 22 '16 at 14:32
  • If you **buy** an IDE, you should be expected to do your own research before making the purchase. The Delphi 10 documentation and sales information clearly states (in the Help->What's New->IDE->Memory Improvements): *In Seattle, the RAD Studio **IDE** includes some notable imporovements related to memory consumption and management: The **IDE** can now use up to 4 GiB of memory, it is no longer limited to 2 GiB.* – Ken White Jan 22 '16 at 14:41
  • 1
    @ventiseis Another thought occurs to me. I know I've been suggesting that perhaps the compiler is not large address aware. But there is one strong indicator that it is. Namely that the IDE contains the compiler. The IDE has the compiler linked directly into it. And when you compile a project in the IDE, under default in process compilation settings, that compilation happens inside bds.exe, the IDE. Since the IDE is large address aware, then surely the compiler code is too. So surely dcc32 will be. – David Heffernan Jan 22 '16 at 14:43
  • I mispoke. The compiler code is not in the bds.exe executable, but in DLLs. However, the conclusion is the same. Those DLLs live happily in a large address aware process and so presumably that same code is also find when hosted by dcc32.exe. – David Heffernan Jan 22 '16 at 15:03
  • 1
    Guys I'm having the same issue try to migrate from XE3 to Berlin. We have a project that takes more then 2G of memory when building. It builds fine on the IDE, but crashes with msbuild. I've enabled /largeaddressaware for dcc32.exe and it seems to have resolved the issue. @ventiseis Have you managed to resolve the issue some other way? – Bruno Santos Mar 13 '17 at 18:21
  • @BrunoSantos Well, the [bug](https://quality.embarcadero.com/browse/RSP-13585) is still marked unresolved. I used a third party tool to add the flag to the binary. We don't use the compiled executables in production, just for automated tests, but so far we didn't encounter any problems with the compiled files. – ventiseis Mar 13 '17 at 19:40

1 Answers1

2

I suspect that this is simply an oversight on the part of Embarcadero. The main focus of the large address aware changes that they made were the IDE. Typically it was the IDE that was susceptible to out of memory failures. I suspect that it is very rare to encounter such failures for the command line compiler.

Now, the compiler code is executed in the IDE process, bds.exe. Under default settings, when you compile a project, an external process is not created to perform the compilation. Instead, if you look at the modules loaded by the IDE, you will see dcc32230.dll and dcc64230.dll. The DLLs contain the 32 and 64 bit Windows compilers. They contain the same code as dcc32.exe and dcc64.exe.

Since the compiler code for dcc32 and dcc64 can exist happily in bds.exe, a large address aware process, it is hard to imagine that dcc32.exe and dcc64.exe will not function under large address aware conditions. Therefore I believe it to be a simple oversight on the part of Embarcadero that these executables do not ship with the large address aware PE flag enabled. I suggest that you submit a bug report / feature request.

So I think it very likely that you are safe to modify the executable files for dcc32.exe and dcc64.exe to enabled the large address aware PE flag. When you do this, you would be wise to perform some top down allocation testing to stress test the change. Details can be found here: https://stackoverflow.com/a/545097/505088.

Community
  • 1
  • 1
David Heffernan
  • 601,492
  • 42
  • 1,072
  • 1,490