Roughly, different operating systems read executables in different binary formats and these don't happen to be compatible. So, when you are building the game from source code, you're building for a given platform, using the formats of that platform. You are correct in guessing that the platform you're building for (also called the target platform) doesn't have to be the platform you're building on (this process is called cross-compiling).
Additionally, it's often the case that some source code isn't translatable to all platforms. For example, code that calls to a library file that's only available on Windows can't be compiled (or cross-compiled) for non-Windows systems. The way that every operating system creates user interfaces (such as windows, touch screen pages, etc) is different not only at the binary level, but often also at the source-code level.
The problem of the second paragraph can be partially mitigated by cross-platform libraries (like GTK, OpenGL, OpenCL, etc) that offer the same mechanisms regardless of the operating system you're targeting. They often achieve this by using conditional compilation and other "tricks" behind the scenes.
The problem of the first paragraph is more difficult, but workarounds have appeared as well, utilizing some other binary formats (Java archives, CLR assemblies, etc). These formats aren't special in any way. The trick is that they aren't meant to be directly read by any operating system - instead, they are fed to another mechanism (the JVM, the CLI, etc) which has already been built by someone multiple times for multiple platforms (including Windows and Linux) and that mechanism manages the execution of those programs, often from a common binary file without the need for you to target any one specific platform. They usually achieve this by automatically compiling the code for you just before it runs (a process known as Just-in-Time compilation).