0

Sounds like a stupid question, but I don't know how else to ask it: I am trying to make a python library that will test CPUID codes and return a result, that happens to work on both linux and windows. The only way I could figure to do CPUID efficiently was to program it in assembler, so, with that said, is there a means by which to test the success or failure of a program run on a machine independant of OS and build it into a python extensible library or, rather, a means by which to build an that is OS independant python library?

Python documentation isn't clear on how this would should/would work. Do I need to make two libraries, one windows and one linux (I'm just trying for one for efficiency)?

NationWidePants
  • 447
  • 8
  • 33
  • Maybe use [inline assembler](http://stackoverflow.com/a/4474704/10077)? – Fred Larson Jun 29 '16 at 16:00
  • [Can't](https://msdn.microsoft.com/en-us/library/4ks26t93.aspx), the issue is windows uses proprietary language for their compilers. I need the windows libraries, and I don't know enough about windows assembly to do this. (note the inline is under a visual studio '15 heading) – NationWidePants Jun 29 '16 at 16:03
  • @Fred Larson just need some assistance/knowledge/enlightenment on cross compiling – NationWidePants Jun 29 '16 at 16:08
  • 3
    You need to make separate Python extension modules for Windows and Linux. You can build them from the same source however. I'd conditionalize (eg. with `#ifdef`) the source code to use the `__cpuid` intrinsic on Windows with the Visual C++ compliler, and GNU-style inline assembler on Linux with GCC or Clang. – Ross Ridge Jun 29 '16 at 16:08
  • @RossRidge Ok, I was afraid that was the answer (Sad panda). The code is already writen for that. Thank you :) – NationWidePants Jun 29 '16 at 16:09
  • I'm not sure why you're sad then. If you already have code that does what I suggest and it works then you don't seem to have a problem. Writing it as an external assembly file isn't going to make it more efficient. – Ross Ridge Jun 29 '16 at 16:41
  • @RossRidge no, but I'll have two libraries instead of one: one for windows and one for linux. I was hoping for one library to rule them all. – NationWidePants Jun 29 '16 at 16:43
  • Something like this: https://github.com/flababah/cpuid.py ? – Elijan9 Jun 30 '16 at 10:49
  • @Elijan9 CPUID is simple in comparison to actually using the SIMD instruction sets. CPUID will only check if I have the hardware to run specific instruction sets, but it is far more complex to write all the assembly for each instruction, as they did in your link, for both linux and windows. Then the question becomes is it more efficient to just write the code as I have already started, and I would assume (as an educated guess) that the C extension will be faster than the python calls to various C extensions. – NationWidePants Jun 30 '16 at 12:02
  • The time it takes to compute prime numbers is far too slow, currently, but with this module it will be lightening fast (up to 2^256, for now, as I am still working on it). – NationWidePants Jun 30 '16 at 12:05
  • Fun fact: there *is* a portable MASM-syntax assembler, called [JWasm](https://github.com/JWasm/JWasm). The question doesn't seem to be specific to MASM, though, so choosing a portable assembler like NASM or GNU in the first place seems like a better bet, so you can assemble the same asm into Windows libraries or Linux libraries. – Peter Cordes Jun 30 '16 at 18:50

1 Answers1

5

Since I don't have the rep to write a comment, I'll have to make an answer.

The reason you can't have "one (dynamic) library to rule them all" is because, Windows and Linux consider dynamic lib files (.dll's, .so's) to be executable formats. Windows dynamic libs have the PE header, which Linux cannot execute, and vice versa, Linux shared libs must have the ELF header, which Windows cannot execute. It doesn't matter if your code is 100% platform independent or not.

As for a static lib, it should be possible to make one that is translatable both by Windows and Linux. Failing that, you could assemble your platform independent "lib" to a common format such as COFF or BIN and link it on the platform you are currently using. Maybe even use some form of runtime-linking.

I'm sorry I used the answer section, without providing the specifics. But if you are simply interested in making a static lib, which will be inlined in your code, on both Linux and Windows, it should be possible.

EDIT: After doing some research I found out, that it is not possibly to use MASM to assemble a flat binary. Apparently it is against the EULA to make one? Regardless the process described earlier using common format, would not work with MASM as it produces PE/COFF as the "lowest level" object files.

So the solution could be to; 1) Port your assembly to something like NASM (netwide assembler), which can produce flat binaries.

2) Strip the PE header to extract the flat binary, using QuasarDonkey's VERY INVOLVED method: https://stackoverflow.com/a/26877436/6509761

Regardless of how you get the flat binary you would have to;

On Linux: Link the flat binary as an ELF executable, should be pretty straight forward.

FLAT BINARY -> ELF EXECUTABLE

On Windows: Add the PE/COFF header to the the flat binary, and link it into an .exe/.dll file. Microsoft has this process, pretty much, on lock down. I couldn't find any examples on how to do this. But theoretically the process would look like this.

FLAT BINARY --> PE/COFF -> WINDOWS EXECUTABLE

QuasarDonkey's method MIGHT be an option for the determined (which is why I included it), but I would never recommend it.

Really, the only straight forward option is to compile a library for Windows and one for Linux, which is unpractical using MASMs PE/COFF format as a base :)

Community
  • 1
  • 1
slowvomit
  • 51
  • 4
  • 1
    Yes, Linux dynamically linked aka shared libraries are [ELF "shared objects"](http://wiki.osdev.org/ELF), hence the `.so` file name. Terminology: what you're calling "raw binaries" are often called "flat" binaries. But yes, NASM's `.bin` output format assembles machine code into the output file with no headers. – Peter Cordes Jun 30 '16 at 18:46
  • 2
    As far as what the OP should actually do: Use NASM or AT&T (GNU) syntax, with assembler macros to detect Windows vs. non-Windows and choose the right function prologue to match the calling convention. It shouldn't be too hard to have one asm source file from which you can produce a Windows DLL or a Linux shared lib, without going through any nonsense like a flat binary with no symbols (which is a problem if there's more than one function, or any non-immediate constants.) – Peter Cordes Jun 30 '16 at 20:08
  • I have updated the my answer with the information you provided in your first post. – slowvomit Jul 01 '16 at 10:06
  • @Peter Cordes I don't know if that would work, I think you would get a segment error if you attempted to make an external binary that is called by a python object. You get strange errors and compile issues with "things" like that, when it comes to python extensions. – NationWidePants Jul 01 '16 at 10:20
  • 1
    You suggest the OP use a portable assembler, and I couldn't agree more. NASM has some projects, which use macros to generate PE: http://forum.nasm.us/index.php?topic=2117.0. However, I was more focused on answering the OPs notion on having "one library to rule them all", as in one compiled library, which would work in both Windows and Linux space, using MASM. This is not practical because PE/COFF requires the initial two bytes to be 'M' 'Z', and ELF requires the initial 4 bytes to be '\x7f' 'E' 'L' 'F'. You simply can't have both. So I offered a unpractical and absolutely non-sane solution. – slowvomit Jul 01 '16 at 10:28
  • @slowvomit: yeah, it's a good and interesting answer to that part of the question. I just wanted to make sure it was clear that it's a bad idea, and the "normal" way to do things is portable source with some `#ifdef`-type stuff, but platform-specific binaries :P – Peter Cordes Jul 01 '16 at 14:49