Building Python C module on Windows

Question

I am trying to build a 'C' python extension on Windows, the core C code compiles absolutely fine, but I am unable to build the python module using setuptool as I am getting

mandlebrot.c(36): fatal error C1083: Cannot open include file: 'stdio.h': No such file or directory

error: command 'e:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.35.32215\bin\HostX86\x86\cl.exe' failed with exit code 2

I am trying to build using:

python3.9 setup.py build_ext --inplace --plat-name win32

Any help would be welcome as I have spent a long time trying to work out how to get this to work.

The setup file:

from setuptools import setup, Extension
from subprocess import getoutput

LIB_DIRS=['.\mandlebrot_set\libs']
INCLUDE_DIRS=['.\mandlebrot_set\includes']
    
module1 = Extension("mandlebrot",
                    sources = ["mandlebrot.c", "mandlebrot_python.c"],
                    libraries = ['mpfr', 'mpir'],
                    library_dirs = LIB_DIRS,
                    include_dirs = INCLUDE_DIRS)

setup(name = "PackageName",
      version = '0.1',
      description = 'Mandlebrot Set calculator',
      author = '',
      author_email = '',
      ext_modules = [module1]
)

Answer 1

1. The error

By default, Python is built on Win with VStudio ([Python.Wiki]: WindowsCompilers), and it also uses that to build C / C++ code (unless otherwise instructed).

It seems like you don't have VStudio's cross build tools (for 032bit) installed. A quick search revealed:

• [SO]: Cannot open include file: 'stdio.h' - Visual Studio Community 2017 - C++ Error

• [MSDN.Social]: how to resolve fatal error C1083: Cannot open include file: 'stdio.h': No such file or directory

My installation (Visual Studio Installer) looks like:

1. Make sure you have installed:

   • The build tools

   • At least one Windows SDK version

2. Try building a simple Hello World application ([MS.DevBlogs]: C++ Tutorial: Hello World) for 032bit (Win32)

   Some VStudio links that might help:

   • [SO]: Visual Studio NMake build fails with: fatal error U1052: file 'win32.mak' not found (@CristiFati's answer)

   • [SO]: LNK2005 Error in CLR Windows Form (@CristiFati's answer)

   • [SO]: Calling Python function with parametrs from C++ project (Visual Studio) (@CristiFati's answer)

   • [SO]: How to include OpenSSL in Visual Studio (@CristiFati's answer)

3. Once the build works, try again the extension

4. If any of the 2 previous steps still fails, you might think of repairing (or even reinstalling - as a last resort) your VStudio installation

2. Setup

Since there are files not provided for a MCVE ([SO]: How to create a Minimal, Reproducible Example (reprex (mcve))), I'm going to use some from [SO]: How to create python C++ extension with submodule that can be imported (@CristiFati's answer) (latest one that I worked on involving extension modules).

I also want to point out setup.py deprecation ([SO]: 'setup.py install is deprecated' warning shows up every time I open a terminal in VSCode), but I'm not going to insist on that.

setup.py:

#!/usr/bin env python

import os
from setuptools import Extension, setup


# @TODO - cfati: Use sources from the other SO question
SOURCE_DIR = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "q076222409")

# Define the extension module
extension_mod = Extension("parent",
                          sources=[os.path.join(SOURCE_DIR, "custom.cc")])

# Define the setup parameters
setup(
    name="parent",
    version="1.0",
    description="A C++ extension module for Python.",
    ext_modules=[extension_mod],
)

I have 2 separate dirs for 064bit (pc064) and 032bit (pc032) builds ⁽¹⁾ (you'll see later why).

3. Build for pc064

This is native (or direct) build, as (CPU and) default Python installation (that I'm going to use) is pc064.

Output:

[cfati@CFATI-5510-0:e:\Work\Dev\StackExchange\StackOverflow\q076249589\pc064]> sopr.bat
### Set shorter prompt to better fit when pasted in StackOverflow (or other) pages ###

[prompt]>
[prompt]> dir /b

[prompt]>
[prompt]> dir /b ..
pc032
pc064
setup.py

[prompt]>
[prompt]> :: Build for pc064
[prompt]> "e:\Work\Dev\VEnvs\py_pc064_03.10_test0\Scripts\python.exe" ..\setup.py build_ext --inplace
running build_ext
building 'parent' extension
creating build
creating build\temp.win-amd64-cpython-310
creating build\temp.win-amd64-cpython-310\Release
creating build\temp.win-amd64-cpython-310\Release\Work
creating build\temp.win-amd64-cpython-310\Release\Work\Dev
creating build\temp.win-amd64-cpython-310\Release\Work\Dev\StackExchange
creating build\temp.win-amd64-cpython-310\Release\Work\Dev\StackExchange\StackOverflow
creating build\temp.win-amd64-cpython-310\Release\Work\Dev\StackExchange\StackOverflow\q076222409
C:\Install\pc064\Microsoft\VisualStudioCommunity\2022\VC\Tools\MSVC\14.35.32215\bin\HostX86\x64\cl.exe /c /nologo /O2 /W3 /GL /DNDEBUG /MD -Ie:\Work\Dev\VEnvs\py_pc064_03.10_test0\include -Ic:\Install\pc064\Python\Python\03.10\include -Ic:\Install\pc064\Python\Python\03.10\Include -IC:\Install\pc064\Microsoft\VisualStudioCommunity\2022\VC\Tools\MSVC\14.35.32215\include -IC:\Install\pc064\Microsoft\VisualStudioCommunity\2022\VC\Tools\MSVC\14.35.32215\ATLMFC\include -IC:\Install\pc064\Microsoft\VisualStudioCommunity\2022\VC\Auxiliary\VS\include "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.22000.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22000.0\\um" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22000.0\\shared" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22000.0\\winrt" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22000.0\\cppwinrt" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um" /EHsc /Tpe:\Work\Dev\StackExchange\StackOverflow\q076
222409\custom.cc /Fobuild\temp.win-amd64-cpython-310\Release\Work\Dev\StackExchange\StackOverflow\q076222409\custom.obj
custom.cc
creating e:\Work\Dev\StackExchange\StackOverflow\q076249589\pc064\build\lib.win-amd64-cpython-310
C:\Install\pc064\Microsoft\VisualStudioCommunity\2022\VC\Tools\MSVC\14.35.32215\bin\HostX86\x64\link.exe /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /LIBPATH:e:\Work\Dev\VEnvs\py_pc064_03.10_test0\libs /LIBPATH:c:\Install\pc064\Python\Python\03.10\libs /LIBPATH:c:\Install\pc064\Python\Python\03.10 /LIBPATH:e:\Work\Dev\VEnvs\py_pc064_03.10_test0\PCbuild\amd64 /LIBPATH:C:\Install\pc064\Microsoft\VisualStudioCommunity\2022\VC\Tools\MSVC\14.35.32215\ATLMFC\lib\x64 /LIBPATH:C:\Install\pc064\Microsoft\VisualStudioCommunity\2022\VC\Tools\MSVC\14.35.32215\lib\x64 "/LIBPATH:C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\lib\um\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.22000.0\ucrt\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\\lib\10.0.22000.0\\um\x64" /EXPORT:PyInit_parent build\temp.win-amd64-cpython-310\Release\Work\Dev\StackExchange\StackOverflow\q076222409\custom.obj /OUT:build\lib.win-amd64-cpython-310\parent.cp310-win_amd64.pyd /IMPLIB:build\temp.win-am
d64-cpython-310\Release\Work\Dev\StackExchange\StackOverflow\q076222409\parent.cp310-win_amd64.lib
   Creating library build\temp.win-amd64-cpython-310\Release\Work\Dev\StackExchange\StackOverflow\q076222409\parent.cp310-win_amd64.lib and object build\temp.win-amd64-cpython-310\Release\Work\Dev\StackExchange\StackOverflow\q076222409\parent.cp310-win_amd64.exp
Generating code
Finished generating code
copying build\lib.win-amd64-cpython-310\parent.cp310-win_amd64.pyd ->

[prompt]>
[prompt]> dir /b
build
parent.cp310-win_amd64.pyd

[prompt]>
[prompt]> :: Try importing in Python pc064
[prompt]> "e:\Work\Dev\VEnvs\py_pc064_03.10_test0\Scripts\python.exe" -c "import sys;print(sys.version);from parent.child import hello;print(hello());print(\"Done.\n\")"
3.10.9 (tags/v3.10.9:1dd9be6, Dec  6 2022, 20:01:21) [MSC v.1934 64 bit (AMD64)]
Hi, World!
Done.

So far, so good.

4. Build for pc032

This is cross build (as host and target architectures don't match).

According to [Python.Docs]: Creating Built Distributions - Cross-compiling on Windows (emphasis is mine):

To cross-compile, you must download the Python source code and cross-compile Python itself for the platform you are targeting - it is not possible from a binary installation of Python (as the .lib etc file for other platforms are not included).

Although it's old, and the other way around it also applies (partially) to our situation.
As explained in the (build related) URLs above, when building for Win, the linker needs the .lib files (${PYTHONCORE}.lib in our case) for the target architecture, so they are required to exist on the host, and also setup.py (SetupTools) needs to find them.

Output (I'll reuse this console):

[cfati@CFATI-5510-0:e:\Work\Dev\StackExchange\StackOverflow\q076249589\pc032]> sopr.bat
### Set shorter prompt to better fit when pasted in StackOverflow (or other) pages ###

[prompt]>
[prompt]> dir /b

[prompt]>
[prompt]> dir /b ..
pc032
pc064
setup.py

[prompt]>
[prompt]> :: Build for pc032
[prompt]> "e:\Work\Dev\VEnvs\py_pc064_03.10_test0\Scripts\python.exe" ..\setup.py build_ext --inplace --plat-name win32 -L"e:\Work\Dev\VEnvs\py_pc032_03.10_test0\libs"
running build_ext
building 'parent' extension
creating build
creating build\temp.win-amd64-cpython-310
creating build\temp.win-amd64-cpython-310\Release
creating build\temp.win-amd64-cpython-310\Release\Work
creating build\temp.win-amd64-cpython-310\Release\Work\Dev
creating build\temp.win-amd64-cpython-310\Release\Work\Dev\StackExchange
creating build\temp.win-amd64-cpython-310\Release\Work\Dev\StackExchange\StackOverflow
creating build\temp.win-amd64-cpython-310\Release\Work\Dev\StackExchange\StackOverflow\q076222409
C:\Install\pc064\Microsoft\VisualStudioCommunity\2022\VC\Tools\MSVC\14.35.32215\bin\HostX86\x86\cl.exe /c /nologo /O2 /W3 /GL /DNDEBUG /MD -Ie:\Work\Dev\VEnvs\py_pc064_03.10_test0\include -Ic:\Install\pc064\Python\Python\03.10\include -Ic:\Install\pc064\Python\Python\03.10\Include -IC:\Install\pc064\Microsoft\VisualStudioCommunity\2022\VC\Tools\MSVC\14.35.32215\include -IC:\Install\pc064\Microsoft\VisualStudioCommunity\2022\VC\Tools\MSVC\14.35.32215\ATLMFC\include -IC:\Install\pc064\Microsoft\VisualStudioCommunity\2022\VC\Auxiliary\VS\include "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.22000.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22000.0\\um" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22000.0\\shared" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22000.0\\winrt" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22000.0\\cppwinrt" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um" /EHsc /Tpe:\Work\Dev\StackExchange\StackOverflow\q076
222409\custom.cc /Fobuild\temp.win-amd64-cpython-310\Release\Work\Dev\StackExchange\StackOverflow\q076222409\custom.obj
custom.cc
creating e:\Work\Dev\StackExchange\StackOverflow\q076249589\pc032\build\lib.win-amd64-cpython-310
C:\Install\pc064\Microsoft\VisualStudioCommunity\2022\VC\Tools\MSVC\14.35.32215\bin\HostX86\x86\link.exe /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /LIBPATH:e:\Work\Dev\VEnvs\py_pc032_03.10_test0\libs /LIBPATH:e:\Work\Dev\VEnvs\py_pc064_03.10_test0\libs /LIBPATH:c:\Install\pc064\Python\Python\03.10\libs /LIBPATH:c:\Install\pc064\Python\Python\03.10 /LIBPATH:e:\Work\Dev\VEnvs\py_pc064_03.10_test0\PCbuild\win32 /LIBPATH:C:\Install\pc064\Microsoft\VisualStudioCommunity\2022\VC\Tools\MSVC\14.35.32215\ATLMFC\lib\x86 /LIBPATH:C:\Install\pc064\Microsoft\VisualStudioCommunity\2022\VC\Tools\MSVC\14.35.32215\lib\x86 "/LIBPATH:C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\lib\um\x86" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.22000.0\ucrt\x86" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\\lib\10.0.22000.0\\um\x86" /EXPORT:PyInit_parent build\temp.win-amd64-cpython-310\Release\Work\Dev\StackExchange\StackOverflow\q076222409\custom.obj /OUT:build\lib.win-amd64-cpython-310
\parent.cp310-win_amd64.pyd /IMPLIB:build\temp.win-amd64-cpython-310\Release\Work\Dev\StackExchange\StackOverflow\q076222409\parent.cp310-win_amd64.lib
   Creating library build\temp.win-amd64-cpython-310\Release\Work\Dev\StackExchange\StackOverflow\q076222409\parent.cp310-win_amd64.lib and object build\temp.win-amd64-cpython-310\Release\Work\Dev\StackExchange\StackOverflow\q076222409\parent.cp310-win_amd64.exp
Generating code
Finished generating code
copying build\lib.win-amd64-cpython-310\parent.cp310-win_amd64.pyd ->

[prompt]>
[prompt]> dir /b
build
parent.cp310-win_amd64.pyd

[prompt]>
[prompt]> :: Success! But its name doesn't seem right. Try importing in Python pc032
[prompt]> "e:\Work\Dev\VEnvs\py_pc032_03.10_test0\Scripts\python.exe" -c "import sys;print(sys.version);from parent.child import hello;print(hello());print(\"Done.\n\")"
3.10.9 (tags/v3.10.9:1dd9be6, Dec  6 2022, 19:43:38) [MSC v.1934 32 bit (Intel)]
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'parent'

[prompt]>
[prompt]> :: Try importing in Python pc064 (this also seems wrong)
[prompt]> "e:\Work\Dev\VEnvs\py_pc064_03.10_test0\Scripts\python.exe" -c "import sys;print(sys.version);from parent.child import hello;print(hello());print(\"Done.\n\")"
3.10.9 (tags/v3.10.9:1dd9be6, Dec  6 2022, 20:01:21) [MSC v.1934 64 bit (AMD64)]
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: DLL load failed while importing parent: %1 is not a valid Win32 application.

Notes:

• The .pyd name is the same as for pc064 case, which is not correct (this is why I used 2 dirs (check note ^#1.), otherwise the pc064 .pyd would have been overwritten). Check:

  • [SO]: What does version name 'cp27' or 'cp35' mean in Python? (@WayneWerner's answer) for naming details

  • [SO]: How do I determine if my python shell is executing in 32bit or 64bit mode on OS X? (@CristiFati's answer)

  • [SO]: Python Ctypes - loading dll throws OSError: [WinError 193] %1 is not a valid Win32 application (@CristiFati's answer) for the encountered error

Things are definitely wrong, could be a SetupTools bug (maybe fixed in later versions), this might not be the case on your side. Simplest way is to rename the .pyd (there might also be a setup.py argument?).

Output (continued):

[prompt]> ren parent.cp310-win_amd64.pyd parent.cp310-win32.pyd

[prompt]>
[prompt]> dir /b
build
parent.cp310-win32.pyd

[prompt]>
[prompt]> :: Try importing in Python pc032
[prompt]> "e:\Work\Dev\VEnvs\py_pc032_03.10_test0\Scripts\python.exe" -c "import sys;print(sys.version);from parent.child import hello;print(hello());print(\"Done.\n\")"
3.10.9 (tags/v3.10.9:1dd9be6, Dec  6 2022, 19:43:38) [MSC v.1934 32 bit (Intel)]
Hi, World!
Done.

Worked like a charm!

This is the most generic way of doing things, and it should work for any target build (pc032, aarch64, arm32), once the cross build tools and the appropriate .lib files are present on the host machine.
But, if aiming for pc032 (win32) only, there's a shortcut:

1. Download and install the pc032 Python (which runs on pc064 machines)

2. Use it (like in #3.) to build natively