Why do Python extension modules need to be compiled with MSVC on Windows?

Question

I have seen it mentioned many times that Python extension modules on Windows must be built with MSVC. It is also mentioned here:

https://docs.python.org/3/extending/windows.html

Windows Python is built in Microsoft Visual C++; using other compilers may or may not work (though Borland seems to). The rest of this section is MSVC++ specific.

I am looking to gain a technical understanding of why this is necessary. What specific things can potentially go wrong when using MinGW-w64? Under what circumstances will things go wrong? Does it matter if the module is written in C or C++? I am looking for a starting point for further research, a high-level overview or best of all: references.

Note that I am not very familiar with development on Windows.

My motivation for asking this question is to understand how big a risk one is taking by compiling with MinGW when things appear to work. Some dependencies may be particularly troublesome to compile with MSVC, but easy to get working with MSYS / MinGW.

There is some discussion of this here, but it's not quite clear to me what he refers to by "object initialization": https://github.com/wbhart/mpir/issues/236#issuecomment-348597754 — Szabolcs, May 19 '19 at 17:44
https://wiki.python.org/moin/WindowsCompilers "MinGW is an alternative C/C++ compiler that works with all Python versions up to 3.4." Since then a different MSVC version has been used. — Antti Haapala -- Слава Україні, May 19 '19 at 18:05
Note that the version needs to be very much the MSVC++ from VS2015 for Python 3.5+ — Antti Haapala -- Слава Україні, May 19 '19 at 18:07
Also http://mingw.org/wiki/Interoperability_of_Libraries_Created_by_Different_Compiler_Brands — Antti Haapala -- Слава Україні, May 19 '19 at 18:08
sorry. 2015 for Python 3.5 and 3.6, and 2017 for [Python 3.7](https://github.com/python/cpython/blob/3.7/PCbuild/readme.txt) — Antti Haapala -- Слава Україні, May 19 '19 at 18:09
Related: https://stackoverflow.com/questions/41975672/abi-compatibility-of-visual-studio-c-libraries — Szabolcs, May 20 '19 at 12:29

score 2 · Answer 1 · answered May 19 '19 at 18:17

2

The reason is that every single release of Microsoft Visual Studio comes with its own runtime, the MSVCRTxx.DLL, there is no system-wide C library as there is in Linux. Since the newer MSVCRT is not freely redistributable it cannot be used by MinGW. Instead MinGW relies on the old MSVCRT.DLL that does not even support most C99 features.

One cannot compatibly link code compiled with different versions of MSVC++ (using different runtimes), so it is inevitable that MinGW-produced code that is mostly MSVC6.0 compatible cannot be linked together with Python compiled with the C/C++ compiler from VS 2015 or 2017.

More information here.

answered May 19 '19 at 18:17

Antti Haapala -- Слава Україні

129,958
22
279
321

3

msvcrt.dll is a private C runtime library that's used by many Windows system executables and shared libraries (491 and 2,156 respectively when last I checked in Windows 10 1803). The 7.x version number of msvcrt.dll indicates it has been in maintenance mode since Windows 7. It's gradually being replaced by the Universal CRT (ucrtbase.dll), which uses API sets such as [api-ms-win-crt-convert-l1-1-0.dll](https://learn.microsoft.com/en-us/uwp/win32-and-com/win32-extension-apis#apis-from-api-ms-win-crt-convert-l1-1-0dll). – Eryk Sun May 20 '19 at 02:09
3

Windows Python 3.5+ uses the Universal CRT, plus a small amount of code that's tightly coupled to MSVC in vcruntime140.dll. I haven't followed what the issues are that prevent updating MinGW to use the Universal CRT and thus make it possible to use MinGW for building extension modules in 3.5+. – Eryk Sun May 20 '19 at 02:12
@eryksun you should add an answer – Antti Haapala -- Слава Україні May 20 '19 at 06:15
1

"One cannot compatibly link code compiled with different versions of MSVC" This is not very specific. I would like to see a specific breakage so I could have a better understanding of when it will or it won't occur. – Szabolcs May 20 '19 at 10:10
1

Let me give an example: if you allocate memory with `malloc` from one library and try to free it with `free` from another, that would clearly cause trouble. However, this is also specific enough that one can check if whether it applies to a project or not. I have successfully linked MSVC-compiled and MinGW compiled libraries without problems (the one problem I needed to deal with was exactly the malloc/free issue). – Szabolcs May 20 '19 at 10:10
1

More generally, it could be an issue if one tries to use groups of functions which act on the same data structure. If creating the data structure with one version, it may not be possible to use it with another. As far as I can see, in the standard C library this occurs with malloc/free and `FILE *` pointers. Are there other issues one needs to pay attention to? – Szabolcs May 20 '19 at 10:13
@Szabolcs I do not know any details, I just added what I know. In any case, the `FILE` structure is a rather serious one as there are stuff in `` that can be a macro, for example. – Antti Haapala -- Слава Україні May 20 '19 at 10:30
@AnttiHaapala Indeed, the `FILE *` thing would be a serious issue. However, many libraries never deal with files at all (think scientific numerical stuff), nor do they mix in- and out-of-library memory allocation / deallocation. Some libraries are a huge pain to build with MSVC, especially for someone not experienced with Windows. However, they are easy to build with msys/mingw (maybe they use a ` ./configure` script). My motivation for asking this question is to understand how big a risk one is taking by compiling with MinGW when things *appear* to work. – Szabolcs May 20 '19 at 10:35

Why do Python extension modules need to be compiled with MSVC on Windows?

1 Answers1