63

I want to wrap a test project containing C++ and OpenMP code with Cython, and build it with distutils via a setup.py file. The content of my file looks like this:

from distutils.core import setup
from distutils.extension import Extension
from Cython.Build import cythonize
from Cython.Distutils import build_ext


modules = [Extension("Interface",
                     ["Interface.pyx", "Parallel.cpp"],
                     language = "c++",
                     extra_compile_args=["-fopenmp"],
                     extra_link_args=["-fopenmp"])]

for e in modules:
    e.cython_directives = {"embedsignature" : True}

setup(name="Interface",
     cmdclass={"build_ext": build_ext},
     ext_modules=modules)

The -fopenmp flag is used with gcc to compile and link against OpenMP. However, if I just invoke

cls ~/workspace/CythonOpenMP/src $ python3 setup.py build

this flag is not recognized, because the compiler is clang:

running build
running build_ext
skipping 'Interface.cpp' Cython extension (up-to-date)
building 'Interface' extension
cc -Wno-unused-result -fno-common -dynamic -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -I/usr/local/include -I/usr/local/opt/sqlite/include -I/usr/local/Cellar/python3/3.3.0/Frameworks/Python.framework/Versions/3.3/include/python3.3m -c Interface.cpp -o build/temp.macosx-10.8-x86_64-3.3/Interface.o -fopenmp
clang: warning: argument unused during compilation: '-fopenmp'
cc -Wno-unused-result -fno-common -dynamic -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -I/usr/local/include -I/usr/local/opt/sqlite/include -I/usr/local/Cellar/python3/3.3.0/Frameworks/Python.framework/Versions/3.3/include/python3.3m -c Parallel.cpp -o build/temp.macosx-10.8-x86_64-3.3/Parallel.o -fopenmp
clang: warning: argument unused during compilation: '-fopenmp'
Parallel.cpp:24:10: warning: unknown pragma ignored [-Wunknown-pragmas]
        #pragma omp parallel for
                ^
1 warning generated.
c++ -bundle -undefined dynamic_lookup -L/usr/local/lib -L/usr/local/opt/sqlite/lib build/temp.macosx-10.8-x86_64-3.3/Interface.o build/temp.macosx-10.8-x86_64-3.3/Parallel.o -o build/lib.macosx-10.8-x86_64-3.3/Interface.so -fopenmp
ld: library not found for -lgomp
clang: error: linker command failed with exit code 1 (use -v to see invocation)
error: command 'c++' failed with exit status 1

I've unsucessfully tried to specify gcc:

cls ~/workspace/CythonOpenMP/src $ python3 setup.py build --compiler=g++-4.7
running build
running build_ext
error: don't know how to compile C/C++ code on platform 'posix' with 'g++-4.7' compiler

How can I tell distutils to use gcc?

clstaudt
  • 21,436
  • 45
  • 156
  • 239

6 Answers6

59

Try setting the "CC" environment variable from inside the setup.py with os.environ.

timotimo
  • 4,299
  • 19
  • 23
  • 27
    `os.environ["CC"] = "g++-4.7" os.environ["CXX"] = "g++-4.7"` just worked – clstaudt May 24 '13 at 14:45
  • 7
    This does not work for me. My problem is that `setup.py` wants to use `icc`. When I set `CC=gcc`, it does try to use `gcc`, but it keeps using the command line arguments appropriate for `icc`, among them `-fp-model strict` which `gcc` doesn't understand, and aborts. So it looks like just setting `CC` is not the right way to do this. – amaurea Jun 18 '15 at 15:54
  • 2
    Distutils does *not* check for CXX, so setting that may end up confusing you. You might want to remove the suggestion to add "CXX" since it is misleading. – SethMMorton Nov 22 '16 at 19:10
  • 1
    `psutil.py` was partly ignoring the `CC` setting so I found a *really* ugly way to build it on Solaris 11: `ln -s /usr/bin/gcc /usr/bin/cc`. – Philip Kearns Mar 13 '17 at 10:48
  • 2
    I also found that the CC setting was partially ignored – Lydia Duncan Jun 27 '18 at 15:57
23

I just took a look at the distutils source, and the --compiler option expects "unix", "msvc", "cygwin", "mingw32", "bcpp", or "emx". It checks the compiler name you want by checking the CC environment variable. Try calling build like this:

CC=gcc python setup.py build

You don't need to set CXX, it doesn't check for that.

SethMMorton
  • 45,752
  • 12
  • 65
  • 86
  • On ubuntu 12.04 it really doesn't look at the CXX environmental variable, that is not what one would expect. (might be the case that it doesnt check on other platforms either ofcourse) – hetepeperfan Sep 20 '16 at 15:59
  • @hetepeperfan I'm having a hard time understanding your comment. Are you agreeing or disagreeing with this answer? – SethMMorton Sep 20 '16 at 16:03
  • 2
    Well in build systems like the autotools, CMake (which generate Makefiles) and Makefiles. They inspect the CC and CXX environmental variables. Then in the makefile they execute `$(CC) -c csource.c` or `$(CXX) -c cppsource.cpp` to create objectfiles. And apparently distunitils break with this conventions. This is why I upvoted your answer because the phrase "you don't need to set CXX" solved my problem, but many people using those mentioned build tools will expect that CC is for building C and CXX for building C++ programs. – hetepeperfan Sep 20 '16 at 18:52
  • I see what you were trying to say. Yes, it is confusing indeed. I'm not sure the original authors were thinking things through in general with distutils. – SethMMorton Sep 20 '16 at 19:04
  • in macOS I also had to set `CXX=g++-9` in order to get it to work. Otherwise it was using clang++. – rvimieiro Jan 22 '20 at 22:38
22

Just in case some others are facing the same problem under Windows (where CC environment variable wouldn't have any effect) :

  • Create file "C:\Python27\Lib\distutils\distutils.cfg" and write this inside :

Code :

[build]
compiler = mingw32
  • Remove all instances of "-mno-cygwin" gcc option from file "C:\Python27\Lib\distutils\cygwinccompiler.py" :

This :

    self.set_executables(compiler='gcc -mno-cygwin -O -Wall',
                         compiler_so='gcc -mno-cygwin -mdll -O -Wall',
                         compiler_cxx='g++ -mno-cygwin -O -Wall',
                         linker_exe='gcc -mno-cygwin',
                         linker_so='%s -mno-cygwin %s %s'
                                    % (self.linker_dll, shared_option,
                                       entry_point))

Becomes this :

self.set_executables(compiler='gcc -O -Wall',
                     compiler_so='gcc -mdll -O -Wall',
                     compiler_cxx='g++ -O -Wall',
                     linker_exe='gcc',
                     linker_so='%s %s %s'
                                % (self.linker_dll, shared_option,
                                   entry_point))

The second point can be necessary in case you are using a recent version of gcc, where the deprecated option -mno-cygwin has been removed.

Hope this will help even if it is not directly related to the OP real needs (but still related to the question's title...)

Gauthier Boaglio
  • 10,054
  • 5
  • 48
  • 85
  • 1
    I was actually struggling with this problem a few minutes ago but I am now clear Thanks man! and why are we adding a new file to the distutils? – thwildfire May 25 '13 at 01:40
  • It's 2014 now, and I have GCC version 4.3.3 and Python verison 3.4. I've implemented your solution with no results. I get the same error. I wrote `[build] compiler = mingw32` into the config file, and deleted `-mcygwin` ... – the_prole Nov 26 '14 at 13:33
  • My answer might be outdated. That is completely possible. I am sorry if this is the case, but it's been a long time I haven't been working on portability stuffs under Windows (strict "Linux work" for months), and actually won't be able to help you with this, right now... Make sure you created `distutils.cfg` at the right place (the requirements might have changed for this file). Have you tried to compile using directly `python3 setup.py build --compiler=mingw32`? – Gauthier Boaglio Nov 26 '14 at 17:42
4

According to this wiki, Python versions after 3.4 do not support MinGW anymore. CPython 3.7 for Windows is compiled with MSC v.1916. When I try to use above-mentioned method with distutils.cfg, I then get an error from distutils: Unknown MS Compiler Version 1916. Looks like it has a hardcoded table of msvcr libraries in its cygwincompiler.py file (which is also responsible for MinGW), and last version known to that file is 1600 from VS2010 / MSVC 10.0.

MarSoft
  • 3,555
  • 1
  • 33
  • 38
1

Try this: http://mail.python.org/pipermail/distutils-sig/2002-August/002944.html

In short, it appears that you should try: python setup.py build --compiler=g++ first.

Andrew W
  • 4,530
  • 1
  • 16
  • 16
0

On linux while using distutils.ccompiler do os.environ('CC')='gcc' and then call distutils.sysconfig.customize_compiler(compiler)

It will do the job.

Rohit T.
  • 41
  • 2