74

This is more of a theoretical question than anything. I'm a Comp sci major with a huge interest in low level programming. I love finding out how things work under the hood. My specialization is compiler design.

Anyway, as I'm working on my first compiler, things are occurring to me that are kind of confusing.

When you write a program in C/C++, the traditional thing people know is, a compiler magically turns your C/C++ code into native code for that machine.

But something doesn't add up here. If I compile my C/C++ program targeting the x86 architecture, it would seem that the same program should run on any computer with the same architecture. But that doesn't happen. You need to recompile your code for OS X or Linux or Windows.(And yet again for 32-bit vs 64-bit)

I'm just wondering why this is the case? Don't we target the CPU architecture/instruction set when compiling a C/C++ program? And a Mac OS and a Windows Os can very much be running on the same exact architecture.

(I know Java and similar target a VM or CLR so those don't count)

If I took a best-shot answer at this, I'd say C/C++ must then compile to OS-specific instructions. But every source I read says the compiler targets the machine. So I'm very confused.

NAND
  • 663
  • 8
  • 22
Nassim Assaf
  • 773
  • 5
  • 5
  • 6
    Because the system calls differently implemented in OS X , Linux, and Windows. – NAND May 06 '20 at 20:46
  • 3
    BTW, this is true for any language that compiles to native instructions, like Fortran and Compiled BASIC. – Thomas Matthews May 06 '20 at 23:30
  • 4
    You might want to read about [ABI](https://en.wikipedia.org/wiki/Application_binary_interface) – rkosegi May 07 '20 at 16:59
  • System calls are mostly handled by dynamically linked runtime libraries so that alone doesn't require recompiling. This has more to do with ABI and calling conventions and binary formats. I think it should be possible to compile/assemble/achieve on one OS while link/package/run on the other? – user3528438 May 08 '20 at 05:33
  • 4
    there are countless number of duplicates: [Why do we need to compile for different platforms?](https://stackoverflow.com/q/48235579/995714), [Why are “Executable files” operating system dependent?](https://stackoverflow.com/q/5477483/995714), [Why does a linux compiled program not work on Windows](https://stackoverflow.com/q/32117572/995714), [Can a C executable file run on all operating systems](https://stackoverflow.com/q/32645978/995714)... – phuclv May 08 '20 at 05:35
  • 3
    ... [Is a Linux executable “compatible” with OS X?](https://stackoverflow.com/q/9439436/995714), [why we must recompile a c source code for a different os on the same machine?](https://stackoverflow.com/q/30574728/995714), [Why can't compiled machine code (EXE, PE, APP) work on all platforms?](https://stackoverflow.com/q/36893707/995714), [Is executable file generated after compiling in C can be copied and run on any different OS (UNIX)?](https://stackoverflow.com/q/11924153/995714) – phuclv May 08 '20 at 05:36

7 Answers7

94

Don't we target the CPU architecture/instruction set when compiling a C/C++ program?

No, you don't.

I mean yes, you are compiling for a CPU instruction set. But that's not all compilation is.

Consider the simplest "Hello, world!" program. All it does is call printf, right? But there's no "printf" instruction set opcode. So... what exactly happens?

Well, that's part of the C standard library. Its printf function does some processing on the string and parameters, then... displays it. How does that happen? Well, it sends the string to standard out. OK... who controls that?

The operating system. And there's no "standard out" opcode either, so sending a string to standard out involves some form of OS call.

And OS calls are not standardized across operating systems. Pretty much every standard library function that does something you couldn't build on your own in C or C++ is going to talk to the OS to do at least some of its work.

malloc? Memory doesn't belong to you; it belongs to the OS, and you maybe are allowed to have some. scanf? Standard input doesn't belong to you; it belongs to the OS, and you can maybe read from it. And so on.

Your standard library is built from calls to OS routines. And those OS routines are non-portable, so your standard library implementation is non-portable. So your executable has these non-portable calls in it.

And on top of all of that, different OSs have different ideas of what an "executable" even looks like. An executable isn't just a bunch of opcodes, after all; where do you think all of those constant and pre-initialized static variables get stored? Different OSs have different ways of starting up an executable, and the structure of the executable is a part of that.

Nicol Bolas
  • 449,505
  • 63
  • 781
  • 982
  • Thank you so much for your answer! That helped alot! This stuff is so fascinating. If I may ask one last thing, if an executable contains OS calls, how does the OS know how to print stuff? Eventually doesn't it have to ask the CPU to do it, on your behalf? This may sound very dumb, and I may need to research alot more, so don't laugh XD. – Nassim Assaf May 06 '20 at 20:50
  • 1
    The OS isn't something distinct from the CPU; it's just *code*. But it's not code that *you wrote*, so it's irrelevant to your question. – Nicol Bolas May 06 '20 at 20:51
  • 20
    ABI varies also – David Heffernan May 06 '20 at 21:46
  • 7
    @NassimAssaf The OS pokes bytes into GPU RAM to tell it the pixels of the characters you wanted to print. (simplified) – Boann May 07 '20 at 01:44
  • 6
    It's already been hinted to in the final paragraph, yet it's something that deserves an explicit mention: Languages generally need a language support library. That library is responsible, e.g. to initialize objects with static storage duration (as noted in the final paragraph), or to map C++ exceptions to OS- and compiler-specific infrastructure. The language support library is compiler- and OS-specific. – IInspectable May 07 '20 at 09:04
  • 5
    You touched on this, but didn't explicitly mention the *linker* which is a fairly important part of the "compilation" process as most people really mean "build an executable binary" when they say "compile" and not just the compilation step. The linker is also insanely environment-specific even after the compiler does its environment-specific work. – Christopher Schultz May 07 '20 at 17:02
  • 5
    @NassimAssaf You are starting to ask questions which are a little off-topic for this question, but they are good questions! The OS knows about all of the hardware connected to the computer, so it knows that to print words, for example, it needs to interface with some display. It's got a device-driver for that display and it says "hey driver, print this stuff" and the driver does it. So even the OS isn't responsible for "simple" things like printing "hello world". Basically, it's turtles all the way down. – Christopher Schultz May 07 '20 at 17:04
  • 1
    Wouldn't it be possible that the standard library gets shipped with the OS, and then the loader dynamically links it when the program is started? – Bergi May 07 '20 at 20:23
  • 1
    @Bergi that's how it's often done (e.g. glibc on Linux systems), but different OSes have different standards for loading executables and different, not entirely compatible standard libraries. – Peteris May 07 '20 at 21:20
  • @NassimAssaf Now go read about [wine](https://en.wikipedia.org/wiki/Wine_(software)) which _does_ "simply" run windows binaries on linux, by intercepting these custom windows OS calls. – pipe May 07 '20 at 21:35
  • _"And those OS routines are non-portable, so your standard library implementation is non-portable. So your executable has these non-portable calls in it."_ -- well, yeah. But isn't the point of the standard library that it encapsulates those unportable fiddlybits so that the actual C application doesn't need to bother? So the problem can't be within the library. Getting to call the library is another thing of course, and in a way has similar portability issues as calling the OS, and it might be worth mentioning that explicitly. – ilkkachu May 08 '20 at 09:17
  • @ilkkachu I haven't done C/C++ in over a decade, but last I did there was the option of "static linking" and "dynamic linking". Static linking takes the standard library and actually copies it inside your final executable (well, discarding the bits that aren't used). Dynamic linking loads it at runtime from an outside source (.dll on Windows, .so on Linux). Static linking obviously puts non-portable calls inside the binary. But even the dynamic linking part still depends on the "load library" call which is non-portable. And you also need the EXACT version of the external library. – Vilx- May 08 '20 at 10:24
  • @Vilx-, yes, that's my point. getting to call the library is a problem. be it with static or dynamic linking. Knowing _how_ to call it (the ABI) is another. – ilkkachu May 08 '20 at 11:38
19

How do you allocate memory? There's no CPU instruction for allocating dynamic memory, you have to ask the OS for the memory. But what are the parameters? How do you invoke the OS?

How do you print output? How do you open a file? How do you set a timer? How do you display a UI? All of these things require requesting services from the OS, and different OSes provide different services with different calls necessary to request them.

Remy Lebeau
  • 555,201
  • 31
  • 458
  • 770
David Schwartz
  • 179,497
  • 17
  • 214
  • 278
  • 1
    You make a valid point, but then if I may ask a slightly off-topic question, how does the OS itself know how to do that? The OS itself eventually has to manage the hardware, therefore doesn't the CPU to some extent need to know how to print? And to allocate? I mean, you're telling the OS to do that on your behalf, via the API. – Nassim Assaf May 06 '20 at 20:44
  • 8
    @NassimAssaf That's what kernal developers get paid big bucks to figure out. – NathanOliver May 06 '20 at 20:45
  • @NassimAssaf: "*The OS itself eventually has to manage the hardware, therefore doesn't the CPU to some extent need to know how to print?*" That's like saying that if you know the letters of the English alphabet, you know how to write a poem. Yes, that's necessary, but it's hardly *sufficient*. – Nicol Bolas May 06 '20 at 20:49
  • 1
    @NassimAssaf "*The OS itself eventually has to manage the hardware, therefore doesn't the CPU to some extent need to know how to print?*" - not necessarily. Some of that stuff is handled by the BIOS instead of the CPU. The OS kernel knows how to talk to the BIOS. "*And to allocate?*" - that is not the CPU's job. And what about memory that is installed on a graphics card instead of RAM? Again, not the CPU's job to manage. – Remy Lebeau May 06 '20 at 20:49
  • 2
    @NassimAssaf The CPU has to be capable of issuing commands to the hardware but it doesn't have to have any idea how to do it. It does not need to know what hardware is present, where their registers are, what values to put in those registers to accomplish what functions, and so on. That's the job of the OS software, typically the drivers. I can make my phone communicate poetry, so my phone has to be able to send poetry, but it doesn't have to have any idea what poetry is or what structure of English is. – David Schwartz May 06 '20 at 20:51
  • Thank you guys so much! It's a huge leap going very low level like this, but it's alot of fun and you helped alot! Thanks! – Nassim Assaf May 06 '20 at 20:51
  • 2
    @NassimAssaf there are whole books and classes on this subject, far more detailed than what can be discussed here. – Remy Lebeau May 06 '20 at 20:53
  • @NassimAssaf: A typical CPU will have instructions to read or write 8/16/32/64 bits from/to a specified address. Hardware will often be wired up so that some addresses are connected to "ordinary" memory, so a read will yield whatever value was last written, with no other side effects, but accesses to some other addresses will trigger specialized actions. Some CPUs have "IN" and "OUT" instructions which output a "this is an in or out instruction" signal along with an address, but the main point is that hardware outside the CPU controls the effects of hitting addresses. The CPU doesn't care. – supercat May 07 '20 at 19:50
  • _"All of these things require requesting services from the OS, and different OSes provide different services with different calls necessary to request them."_ -- But those services are evident in the function calls made by the C program. There's actually a standard that defines the library calls a C program can use to allocate memory, print output or open a file. The trouble, of course, is getting to call a library implementing them, but this answer doesn't appear to mention that, at all. – ilkkachu May 08 '20 at 09:05
  • @ilkkachu The boundary between library code and non-library code isn't relevant to this question because it's all compiled together. The issue is how the code requests OS services, whether that takes place in the library or in the application. – David Schwartz May 11 '20 at 00:06
  • @DavidSchwartz, well, I suppose it depends on what level we look at it. As far as I understand, e.g. the C standard says nothing about OS level system calls, it just defines a bunch of functions that the standard library makes available. The same for POSIX, actually. And from the point of view of a C program that calls `malloc()`, `printf()` and whatever, their implementation is not part of the program. – ilkkachu May 11 '20 at 09:02
  • @DavidSchwartz, Besides, e.g. on Linux, the C library is _not_ usually compiled together with a random program. Instead it's a precompiled library, which is then linked (dynamically or not) to the program. The issue is very much in that linking step, and the function call ABI already. – ilkkachu May 11 '20 at 09:02
  • @ilkkachu The library is also in C/C++ and also needs to be recompiled for every OS. – David Schwartz May 11 '20 at 16:59
  • @DavidSchwartz, I don't think it's in any way relevant for a C program how the standard library is implemented. Ok, in a wide sense the library _is_ a C program too, and in that case the OS calls made by the library are an example of why a program would have to be compiled separately for each OS (along with some other modifications, possibly). – ilkkachu May 11 '20 at 17:47
  • @DavidSchwartz, however, I did understand the question as referring to a regular user-level application program. And in that case there's just no need to consider OS calls at all, since getting even as far as to the library is platform-dependent. I think that's important, since otherwise there's the obvious counter-question of "why don't we hide the recompilation-needing OS calls in a library then, and remove the need for recompilation of the main program". (Well, sure, something like that could be done, but it's not that common.) – ilkkachu May 11 '20 at 17:50
  • @ilkkachu Applications make both OS calls and library calls. Many library calls are nothing but thin wrappers around OS calls, often even inlining. This is a complexity that doesn't affect the core answer to the question. – David Schwartz May 11 '20 at 19:10
14

If I compile my C/C++ program targeting the x86 architecture, it would seem that the same program should run on any computer with the same architecture.

It is very true, but there're a few nuances.

Let's consider several cases of programs that are, from C-language point of view, OS-independent.

  1. Suppose all that your program does, from the very beginning, is stress-testing the CPU by doing lots of computations without any I/O.

The machine code could be exactly the same for all the OSes (provided they all run in the same CPU mode, e.g. x86 32-bit Protected Mode). You could even write it in assembly language directly, it wouldn't need to be adapted for each OS.

But each OS wants different headers for the binaries containing this code. E.g. Windows wants PE format, Linux needs ELF, macOS uses Mach-O format. For your simple program you could prepare the machine code as a separate file, and a bunch of headers for each OS's executable format. Then all you need to "recompile" would actually be to concatenate the header and the machine code and, possibly, add alignment "footer".

So, suppose you compiled your C code into machine code, which looks as follows:

offset:  instruction  disassembly
    00:  f7 e0        mul eax
    02:  eb fc        jmp short 00

This is the simple stress-testing code, repeatedly doing multiplications of eax register by itself.

Now you want to make it run on 32-bit Linux and 32-bit Windows. You'll need two headers, here're examples (hex dump):

  • For Linux:
000000 7f 45 4c 46 01 01 01 00 00 00 00 00 00 00 00 00  >.ELF............<
000010 02 00 03 00 01 00 00 00 54 80 04 08 34 00 00 00  >........T...4...<
000020 00 00 00 00 00 00 00 00 34 00 20 00 01 00 28 00  >........4. ...(.<
000030 00 00 00 00 01 00 00 00 54 00 00 00 54 80 04 08  >........T...T...<
000040 54 80 04 08 04 00 00 00 04 00 00 00 05 00 00 00  >T...............<
000050 00 10 00 00                                      >....<
  • For Windows (* simply repeats previous line until the address below * is reached):
000000 4d 5a 80 00 01 00 00 00 04 00 10 00 ff ff 00 00  >MZ..............<
000010 40 01 00 00 00 00 00 00 40 00 00 00 00 00 00 00  >@.......@.......<
000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  >................<
000030 00 00 00 00 00 00 00 00 00 00 00 00 80 00 00 00  >................<
000040 0e 1f ba 0e 00 b4 09 cd 21 b8 01 4c cd 21 54 68  >........!..L.!Th<
000050 69 73 20 70 72 6f 67 72 61 6d 20 63 61 6e 6e 6f  >is program canno<
000060 74 20 62 65 20 72 75 6e 20 69 6e 20 44 4f 53 20  >t be run in DOS <
000070 6d 6f 64 65 2e 0d 0a 24 00 00 00 00 00 00 00 00  >mode...$........<
000080 50 45 00 00 4c 01 01 00 ee 71 b4 5e 00 00 00 00  >PE..L....q.^....<
000090 00 00 00 00 e0 00 0f 01 0b 01 01 47 00 02 00 00  >...........G....<
0000a0 00 02 00 00 00 00 00 00 00 10 00 00 00 10 00 00  >................<
0000b0 00 10 00 00 00 00 40 00 00 10 00 00 00 02 00 00  >......@.........<
0000c0 01 00 00 00 00 00 00 00 03 00 0a 00 00 00 00 00  >................<
0000d0 00 20 00 00 00 02 00 00 40 fb 00 00 03 00 00 00  >. ......@.......<
0000e0 00 10 00 00 00 10 00 00 00 00 01 00 00 00 00 00  >................<
0000f0 00 00 00 00 10 00 00 00 00 00 00 00 00 00 00 00  >................<
000100 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  >................<
*
000170 00 00 00 00 00 00 00 00 2e 66 6c 61 74 00 00 00  >.........flat...<
000180 04 00 00 00 00 10 00 00 00 02 00 00 00 02 00 00  >................<
000190 00 00 00 00 00 00 00 00 00 00 00 00 60 00 00 e0  >............`...<
0001a0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  >................<
*
000200

Now if you append your machine code to these headers and, for Windows, also append a bunch of null bytes to make file size 1024 bytes, you'll get valid executables that will run on the corresponding OS.

  1. Suppose now that your program wants to terminate after doing some amount of calculations.

    Now it has two options:

    1. Crash—e.g. by execution of an invalid instruction (on x86 it could be UD2). This is easy, OS-independent, but not elegant.

    2. Ask the OS to correctly terminate the process. At this point we need an OS-dependent mechanism to do this.

On x86 Linux it would be

xor ebx, ebx ; zero exit code
mov eax, 1   ; __NR_exit
int 0x80     ; do the system call (the easiest way)

On x86 Windows 7 it would be

    ; First call terminates all threads except caller thread, see for details:
    ; http://www.rohitab.com/discuss/topic/41523-windows-process-termination/
    mov eax, 0x172  ; NtTerminateProcess_Wind7
    mov edx, terminateParams
    int 0x2e        ; do the system call
    ; Second call terminates current process
    mov eax, 0x172
    mov edx, terminateParams
    int 0x2e
terminateParams:
    dd 0, 0 ; processHandle, exitStatus

Note that on other Windows version you'd need another system call number. The proper way to call NtTerminateProcess is via yet another nuance of OS-dependence: shared libraries.

  1. Now your program wants to load some shared library to avoid reinventing some wheels.

OK, we've seen that our executable file formats are different. Suppose that we've taken this into account and prepared the import sections for the file targeting each of the target OS. There's still a problem: the way to call a function—the so called calling convention—for each OS is different.

E.g. suppose the C-language function your program needs to call returns a structure containing two int values. On Linux the caller would have to allocate some space (e.g. on the stack) and pass the pointer to it as the first parameter to the function being called, like so:

sub esp, 12 ; 4*2+alignment: stack must be 16-byte aligned
push esp    ;                right before the call instruction
call myFunc

On Windows you'd get the first int value of the structure in EAX, and the second in EDX, without passing any additional parameters to the function.

There are other nuances like different name mangling schemes (though these can differ between compilers even on the same OS), different data types (e.g. long double on MSVC vs long double on GCC) etc., but the above mentioned ones are the most important differences between the OSes from the point of view of the compiler and linker.

Ruslan
  • 18,162
  • 8
  • 67
  • 136
9

No, you are not just targeting a CPU. You are also targeting the OS. Let's say you need to print something to the terminal screen using cout. cout will eventually wind up calling an API function for the OS the program is running on. That call can, and will, be different for different operating systems, so that means you need to compile the program for each OS so it makes the correct OS calls.

Remy Lebeau
  • 555,201
  • 31
  • 458
  • 770
NathanOliver
  • 171,901
  • 28
  • 288
  • 402
5
  1. The standard library and the C-runtime must interact with OS API's.
  2. The executable formats for the different target OS's are different.
  3. Different OS kernels can configure the hardware differently. Things like byte order, stack direction, register use conventions, and probably many other things can be physically different.
jwdonahue
  • 6,199
  • 2
  • 21
  • 43
3

Strictly speaking, you don't need to

Program Loaders

You have wine, the WSL1 or darling, which all are loaders for the respective other OS' binary formats. These tools work so well because the machine is basically the same.

When you create an executable, the machine code for "5+3" is basically the same on all x86 based platforms, however there are differences, already mentioned by the other answers, like:

  • file format
  • API: eg. Functions exposed by the OS
  • ABI: Binary layout etc.

These differ. Now, eg. wine makes Linux understand the WinPE format, and then "simply" runs the machine code as a Linux process (no emulation!). It implements parts of the WinAPI and translates it for Linux. Actually, Windows does pretty much the same thing, as Windows programs do not talk to the Windows Kernel (NT) but the Win32 subsystem... which translates the WinAPI into the NT API. As such, wine is "basically" another WinAPI implementation based on the Linux API.

C in a VM

Also, you can actually compile C into something else than "bare" machine code, like LLVM Byte code or wasm. Projects like GraalVM make it even possible to run C in the Java Virtual Machine: Compile Once, Run Everywhere. There you target another API/ABI/File Format which was intended to be "portable" from the start.

So while the ISA makes up the whole language a CPU can understand, most programs don't only "depend" on the CPU ISA but need the OS to be made work. The toolchain must see to that

But you're right

Actually, you are rather close to being right, however. You actually could compile for Linux and Win32 with your compiler and perhaps even get the same result -- for a rather narrow definition of "compiler". But when you invoke the compiler like this:

c99 -o foo foo.c

You don't only compile (translate the C code to, eg., assembly) but you do this:

  1. Run the C preprocessor
  2. Run the "actual" C compiler frontend
  3. Run the assembler
  4. Run the linker

There might be more or less steps, but that's the usual pipeline. And step 2 is, again with a grain of salt, basically the same on every platform. However the preprocessor copies different header files into your compilation unit (step 1) and the linker works completely differently. The actual translation from one language (C) to another (ASM), that is what from a theoretical perspective a compiler does, is platform independent.

ljrk
  • 751
  • 1
  • 5
  • 21
2

For a binary to work properly (or in some cases at all) there are a whole lot of ugly details that need to be consistent/correct including but probablly not limited to.

  • How the C source constructs like procedure calls, parameters, types etc are mapped onto architecture-specific contstructs like registers, memory locations, stack frames etc.
  • How the results of compilation are expressed in an executable file so that the binary loader can load them into the correct places in the virtual address space and/or perform "fixups" after they are loaded in an arbitary place.
  • How exactly the standard library is implemented, sometimes standard library functions are actual functions in a library, but often they are instead macros, inline functions or even compiler builtins that may rely on non-standard functions in the library.
  • Where the boundary between the OS and the application is considered to be, on unix-like systems the C standard library is considered a core platform library. On the other hand on windows the C standard library is considered to be something that the compiler provides and is either compiled into the application or shipped alongside it.
  • How are other libraries implemented? what names do they use? how are they loaded?

Differences in one or more of these things are why you can't just take a binary intended for one OS and load it normally on another.

Having said that it is possible to run code intended for one os on another. That is essentially what wine does. It has special translator libraries that translate windows API calls into calls that are available on Linux and a special binary loader that knows how to load both windows and Linux binaries.

plugwash
  • 9,724
  • 2
  • 38
  • 51