13

FIPS capable OpenSSL has one limitation - it must load libeay32.dll at fixed address and if loads at any other address, it fails initialization check, so it can't be used in FIPS mode.

So we chose address according to Microsoft's recommendation and on some machines that address from time to time is occupied by various other libraries - like MSVCR120_CLR0400.dll or mscorlib.ni.dll or clr.dll, you get the point.

Is there any way to check if some fixed address + length is taken and ask OS to free that part of memory for me, like rebase those dll's to other memory parts or something like this?

Update:

I've collected information from 20 devices with ListDLLs and there is some pattern what is loaded where, but it is far from well defined. So I've ran some math, found largest gap, where nothing was loaded in it in those 20 logs I had, changed libeay32 base address to somewhere in that gap (gap was ~6 times larger than dll, so I've chosen ~middle of it) and still after couple tries application managed to load something in that gap before libeay32 (to be specific - clrjit.dll, it has base address of 0x10000000, which I think is default), although in application I try to load libeay32 as soon as possible.

Deduplicator
  • 44,692
  • 7
  • 66
  • 118
Giedrius
  • 8,430
  • 6
  • 50
  • 91
  • 1
    This question was asked in the mailing list of openssl: https://groups.google.com/forum/#!topic/mailing.openssl.users/LviBi6rhLIU . Possible solutions where to recompile the dll with /FIXED or put the address to *0x50000000 to 0x6FFF0000* – xanatos Jun 22 '15 at 07:27
  • I use address from that space, it improved situation, but not solved it completely. I've used 0x64880000 as fixed address for libeay32.dll. – Giedrius Jun 22 '15 at 07:35
  • @xanatos - Does the OpenSSL Security Policy allow you to change the address as suggested by Bohm? I think you should look for an answer by Steve Marquess or Dr. Henson. – jww Jun 22 '15 at 08:56

1 Answers1

1

Why don't you combine the hints given:

  • Use /INCLUDE with a symbol from libeay.dll when linking your program to force a static dependency on that library.
  • Compile libeay32.dll with /FIXED so it cannot be relocated.

Thus, it's loaded on loading the executable, before any managed code runs, not sometime later dynamically, so all those relocatable dlls aren't there yet and cannot get in the way.

Deduplicator
  • 44,692
  • 7
  • 66
  • 118
  • I could not find /INCLUDE option for msbuild, I guess it is not available for .NET? – Giedrius Jun 30 '15 at 05:38
  • It's an option for the linker which msbuild invokes. – Deduplicator Jun 30 '15 at 11:14
  • According to msdn: Differences Between C# Compiler and C++ Compiler Output: There are no object (.obj) files created as a result of invoking the C# compiler; output files are created directly. As a result of this, the C# compiler does not need a linker. – Giedrius Jul 01 '15 at 06:05
  • Compile it to `.netmodule`s, and then link those. http://blogs.msdn.com/b/texblog/archive/2007/04/05/linking-native-c-into-c-applications.aspx – Deduplicator Jul 02 '15 at 12:25