Since I don't have the rep to write a comment, I'll have to make an answer.
The reason you can't have "one (dynamic) library to rule them all" is because, Windows and Linux consider dynamic lib files (.dll's, .so's) to be executable formats. Windows dynamic libs have the PE header, which Linux cannot execute, and vice versa, Linux shared libs must have the ELF header, which Windows cannot execute. It doesn't matter if your code is 100% platform independent or not.
As for a static lib, it should be possible to make one that is translatable both by Windows and Linux. Failing that, you could assemble your platform independent "lib" to a common format such as COFF or BIN and link it on the platform you are currently using. Maybe even use some form of runtime-linking.
I'm sorry I used the answer section, without providing the specifics. But if you are simply interested in making a static lib, which will be inlined in your code, on both Linux and Windows, it should be possible.
EDIT:
After doing some research I found out, that it is not possibly to use MASM to assemble a flat binary. Apparently it is against the EULA to make one? Regardless the process described earlier using common format, would not work with MASM as it produces PE/COFF as the "lowest level" object files.
So the solution could be to;
1) Port your assembly to something like NASM (netwide assembler), which can produce flat binaries.
2) Strip the PE header to extract the flat binary, using QuasarDonkey's VERY INVOLVED method: https://stackoverflow.com/a/26877436/6509761
Regardless of how you get the flat binary you would have to;
On Linux: Link the flat binary as an ELF executable, should be pretty straight forward.
FLAT BINARY -> ELF EXECUTABLE
On Windows: Add the PE/COFF header to the the flat binary, and link it into an .exe/.dll file. Microsoft has this process, pretty much, on lock down. I couldn't find any examples on how to do this. But theoretically the process would look like this.
FLAT BINARY --> PE/COFF -> WINDOWS EXECUTABLE
QuasarDonkey's method MIGHT be an option for the determined (which is why I included it), but I would never recommend it.
Really, the only straight forward option is to compile a library for Windows and one for Linux, which is unpractical using MASMs PE/COFF format as a base :)