0

I'm trying to compile a large codebase written for a 32 bit embedded processor to run on a 64 bit desktop processor for simulation/unit testing. I need the resulting object to be a shared library. This is not an issue in Windows, I can build a dll like this (/DWIN32) and it runs fine.

In Linux, I'm able to compile and link with the -m32 option given to both gcc and the linker and get a shared library out. The problem is, this library is (just like I specified with -m32) a 32 bit library and will not run on my 64 bit arch. With Python, I try to load the library (using ctypes.cdll.LoadLibrary())

OSError: out.so: wrong ELF class: ELFCLASS32

I discovered the -mx32 option, and according to the docs this is exactly what I want:

The -mx32 option sets int, long, and pointer types to 32 bits, and generates code for the x86-64 architecture.

So, I pass -mx32 to the compiler and the linker (replacing my -m32 option) and get the following (snipped the output):

/usr/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-linux-gnu/4.8/libstdc++.so when searching for -lstdc++
/usr/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-linux-gnu/4.8/libstdc++.a when searching for -lstdc++
/usr/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-linux-gnu/4.8/32/libstdc++.so when searching for -lstdc++
/usr/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-linux-gnu/4.8/32/libstdc++.a when searching for -lstdc++
/usr/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-linux-gnu/4.8/libstdc++.so when searching for -lstdc++
/usr/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-linux-gnu/4.8/libstdc++.a when searching for -lstdc++
/usr/bin/ld: cannot find -lstdc++
/usr/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libm.so when searching for -lm
/usr/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libm.a when searching for -lm
/usr/bin/ld: skipping incompatible /usr/lib/x86_64-linux-gnu/libm.so when searching for -lm
/usr/bin/ld: skipping incompatible /usr/lib/x86_64-linux-gnu/libm.a when searching for -lm
/usr/bin/ld: skipping incompatible /usr/lib/x86_64-linux-gnu/libm.so when searching for -lm
/usr/bin/ld: skipping incompatible /usr/lib/x86_64-linux-gnu/libm.a when searching for -lm
/usr/bin/ld: cannot find -lm

I get the same results with gcc 4.7 and gcc 4.8. The above output is from gcc 4.8.

I have gcc-4.8-multilib and g++-4.8-multilib installed.

My library path is:

LIBRARY_PATH=/usr/lib/x86_64-linux-gnu:/usr/lib/gcc/x86_64-linux-gnu/4.8/32:/usr/lib/gcc/x86_64-linux-gnu/4.8/

From .bashrc I've specified it like this (below), in despair adding the /4.8/ and /4.8/32/ stuff after reading that the linker would only bind with the libraries that work. 

export LIBRARY_PATH=/usr/lib/$(gcc -print-multiarch):/usr/lib/gcc/x86_64-linux-gnu/4.8/32:/usr/lib/gcc/x86_64-linux-gnu/4.8/

As I mention, this works quite well already on Windows as a dll, I have to believe I'm just missing something. Pointers should be 32 bits and longs should be 32 bits, the whole thing should run on x86_64. -mx32 says it will do exactly that (right?).

Checking one of the objects after compiling in -m32:

$:~/project$ file foo.o
foo.o: ELF 32-bit LSB relocatable, Intel 80386, version 1 (SYSV), not stripped

And checking the same object after compiling with -mx32:

$:~/project$ file foo.o
foo.o: ELF 32-bit LSB relocatable, x86-64, version 1 (SYSV), not stripped

Am I going about this the wrong way? Can I still use my 32-bit shared library with some sort of compatibility layer?

I saw a bug report against gcc 4.7 regarding these link errors... But I didn't see much conclusion from it. This page says that gcc 4.8 is the recommended minimum for x32, so I installed it. Still -- I can't get it to link. -m32 links fine.

Peter Cordes
  • 328,167
  • 45
  • 605
  • 847
skelliam
  • 522
  • 3
  • 11
  • Assuming that you indeed need an `ELF 64-bits` , it looks like the linker failed to find libm.so which is provided by glibc. Can you add `/usr/lib64` in your library path? – alvits Oct 20 '15 at 00:49
  • Thanks for the reply @alvits, but I don't have this directory: `$ cd /usr/lib64 bash: cd: /usr/lib64: No such file or directory` – skelliam Oct 20 '15 at 12:24

2 Answers2

4

-m32 is normal 32-bit mode, and yes most x86-64 GNU/Linux systems have multiarch support with 32-bit libraries that you can install if not already installed.

The -mx32 option is not what you want at all. It compiles code for the x32 ABI, which uses 32-bit addresses with 64-bit registers and instructions. The resulting code is not compatible with 32-bit processors (as it uses 64-bit instructions), but is also not compatible with 64-bit code (which will use 64-bit addresses). So everything in user-space has to be compiled with it in mind, including libc, and even the kernel needs to support x32 user-space and syscalls: https://unix.stackexchange.com/questions/121424/linux-and-x32-abi-how-to-use (i.e. just like supporting 32-bit binaries, x32 is effectively a separate architecture.)

Complicating matters further, the testing you've been doing on Windows hasn't been doing what you think at all. Passing /DWIN32 to the Visual C++ compiler only defines a macro called WIN32 (like -DWIN32 would for GCC); it doesn't make the compiler generate 32-bit binaries. 64-bit Windows executables cannot load 32-bit libraries; the libraries you've been testing with have actually been 64-bit.

If you want to unit-test 32-bit code, you'll need to test it on a fully 32-bit target architecture like -m32. x32 is not such a system. (32-bit code under a 64-bit kernel is fine, though, unless you're worried about only having 2G or 3GiB of address-space usable by user-space in your embedded system, instead of 4GiB under a 64-bit kernel.)

Peter Cordes
  • 328,167
  • 45
  • 605
  • 847
  • Thanks @duskwuff. Can you explain how Visual Studio is able to accomplish what I want? Is there already some sort of abstraction layer that allows 32 bit dlls to run? – skelliam Oct 19 '15 at 21:36
  • It doesn't; your testing methodology on Windows is flawed. `/DWIN32` just defines a macro called "WIN32"; it doesn't actually make Visual Studio build 32-bit code. –  Oct 19 '15 at 21:44
  • Any why does it compile and link in Windows? I guess the real question is why can't I link with mx32? I'll figure out what's broken after that. Or, what's the next approach I should take? @duskwuff – skelliam Oct 19 '15 at 22:34
  • It's compiling as a 64-bit library in Windows. `/DWIN32` doesn't make it 32-bit. If you want to test a 32-bit library, you need to load it in a 32-bit executable; there's no way around that. –  Oct 19 '15 at 23:10
  • Okay. So the only course of action is to work though gcc errors as they arise (despite the fact that I don't have the same errors in Visual Studio), and forget about -m32 (because the end goal is to load it and run it from a 64 bit process). Is there some reason I can't link -mx32 and discover that it doesn't run properly? (At the end of the day I'd rather find out now (thank you) I need to build as native 64-bit but the title of my question is the link errors with -mx32). – skelliam Oct 20 '15 at 01:18
  • Oh boy. I just discovered some reasons what I'm doing is a little bit 'easier' in Windows. 1. Windows uses a LLP64 [data model](https://en.wikipedia.org/wiki/64-bit_computing#64-bit_data_models) while Linux uses a LP64 data model. (meaning anywhere I use 'long', it remains a 32 bit value in Windows but becomes a 64 bit value in Linux) 2. Handles in Windows are [64 bit pointers, but only carry 32 significant bits](https://msdn.microsoft.com/en-us/library/windows/desktop/aa384249%28v=vs.85%29.aspx), thus, they can be truncated without loss of data. – skelliam Oct 20 '15 at 12:54
  • And [here](https://msdn.microsoft.com/en-us/library/windows/desktop/aa384231%28v=vs.85%29.aspx) is described a technique for wrapping a 32 bit dll w/ an out of process COM server and using the COM server to handle calls to/from the 32 bit dll. This sounds ugly. – skelliam Oct 20 '15 at 13:02
  • One more question remains @duskwuff, if I were to get this shared library to link with -mx32, would I be able to use it from a 64 bit process? It sounds like the only implication here might be the case where I pass an address that breaks the 32 bit boundary, then it would be truncated to 32 bits and causing havoc... I might be able to wrap the whole thing with a 64 bit wrapper, detect a 64 bit address that will be truncated, and throw an error. – skelliam Oct 20 '15 at 13:08
  • @skelliam Not directly. `-mx32` directs the compiler to generate an X32 binary, which will not have appropriate metadata allowing it to be loaded by a 64-bit process. –  Oct 20 '15 at 16:36
  • Thanks @duskwuff I'll mark this as the answer, if anyone else comes along there is a ton of good stuff here. – skelliam Oct 20 '15 at 16:42
1

Your initial error, trying to load the library you successfully built, indicates that you are NOT using a 32-bit version of Python, so it can't build the 32-bit shared library.

If you install an i386 (32-bit) version of Python it should be able to load the library. What's more, that is the version of Python you'll need to run on your embedded i386 target.

Note that all the above assumes your "32bit embedded processor" is i386 compatible. If its not (its an ARM or MIPS or some other 32bit embedded processor instead), you'll need to install cross compilers for that target to build your library and an emulator (such as QEMU) to run executables.

Chris Dodd
  • 119,907
  • 13
  • 134
  • 226
  • The end goal will be to load it from a 64 bit version of Matlab. I don't need full emulation of the target. I already have abstraction for the target specific bits like eeprom reads and so forth. And I can do what I want in Windows already. I thought I could build the same code in Linux... It sounds like the answer is, port it correctly, or forget it. – skelliam Oct 19 '15 at 21:40