11

I have a setup.py file that is very similar to the one shown here: https://stackoverflow.com/a/49866324/4080129. It looks like this:

from distutils.core import setup, Extension
from Cython.Build import cythonize
import numpy

sources = ["hs/subfolder/detect.pyx",
           "hs/subfolder/Donline.cpp",
           "hs/subfolder/Handler.cpp",
           "hs/subfolder/Process.cpp",
           "hs/subfolder/Filter.cpp",
           "hs/subfolder/Localize.cpp"]

exts = [Extension(name='hs.detect',
                  sources=sources,
                  extra_compile_args=['-std=c++11', '-O3'],
                  include_dirs=[numpy.get_include()])]

setup(
    ext_modules=cythonize(exts),
    include_dirs=[numpy.get_include()]
)

There's a package with some pure-Python, and a submodule that contains Cython files. The setup.py is in the parent folder, not in the Cython one:

setup.py
hs/
    some_python.py
    subfolder/
        detect.pyx
        Donline.cpp
        ...etc

Now, setup.py correctly compiles all the files module/submodule/file1.cpp etc. and saves the build to build/temp.linux-x86_64-3.6/module/submodule/file1.o . However, just after that, it tries to compile a file called file1.cpp, which doesn't exist (the correct one is module/submodule/file1.cpp, and has already been compiled).

gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Ihs/subfolder -I/[...]/python3.6/site-packages/numpy/core/include -I/[...]/python3.6/site-packages/numpy/core/include -I/[...]/include -I/disk/scratch/miniconda/envs/my_default/include/python3.6m -c Donline.cpp -o build/temp.linux-x86_64-3.6/Donline.o -std=c++11 -O3
gcc: error: Donline.cpp: No such file or directory
gcc: fatal error: no input files
compilation terminated.
error: command 'gcc' failed with exit status 4

I'm very confused, this completely prevents my code from compiling...

Martino
  • 729
  • 6
  • 18
  • 2
    Please share the setup script, also it helps if you provide the project structure on disk (what files in what dirs, w/o contents). – hoefling May 21 '18 at 14:23
  • Done. Thank you! – Martino May 21 '18 at 14:29
  • The setup script looks good to me, also can't reproduce the issue. Maybe do a clean rebuild? `python setup.py clean build` or just remove the `build/` dir. Is the error still there? – hoefling May 21 '18 at 17:13
  • Yes, I tried that already. – Martino May 21 '18 at 17:31
  • I also just tried *not* using the virtualenv, and it's the same. This is really really odd. Could it be the C files calling each other? Or something wrong in the __init__ files? I can't think of a reason... – Martino May 21 '18 at 17:34
  • Wait, how do you include i.e. `SpkDonline.cpp` in other sources, by just filename? Also, it's cleaner to create a header file for each source file and include those instead of the source files. – hoefling May 21 '18 at 17:58
  • The files reference each other by calling the headers. For example detect.pyx includes the line `cdef extern from "SpkDonline.h"`. – Martino May 21 '18 at 18:14
  • I'm a bit suspicious of your module name `herdingspikes.detect` - I wonder if Cython is expecting it to be `herdingspikes.detection_localization.detect`. I haven't tested it though, and I'm not sure why it should give that error – DavidW May 21 '18 at 20:01
  • @DavidW I've tried both – Martino May 24 '18 at 12:09
  • Only other thing I can see here is that you haven't told Cython that `language="c++"`. That's another guess I'm afraid (and I don't see why it'd give _that_ error), but it might be worth a try – DavidW May 24 '18 at 13:40
  • I think I had a previous version of setup.py which had that. Thanks for the effort :) – Martino May 24 '18 at 15:38
  • Just out of curiosity - if you provide the absolute path of `SpkDonline.cpp` in the `sources` list, does the error still emerge for `SpkDonline.cpp`? Also, can you paste the full build log? – hoefling May 24 '18 at 16:24
  • I tried with the full path and it didn't help. The full log is too long to paste at the moment, are you looking for anything in particular? – Martino May 29 '18 at 10:55
  • Can you post an [MCVE please](https://stackoverflow.com/help/mcve). There is not enough information to replicate the issue. – danny Jun 01 '18 at 15:17
  • Just solved, thanks! – Martino Jun 03 '18 at 13:36

1 Answers1

6

It turns out the .pyx file contains a line

# distutils: sources = Donline.cpp Handler.cpp Process.cpp Filter.cpp Localize.cpp

which tells distutils what to compile. I wasn't aware of it, and since it looks an awful lot like a commented-out line, I didn't realise it was there.

Cython tries to compile also these, other than the ones contained in the setup.pyfile, i.e. neither of the two sources list overrides the other. Apparently, these sources, despite being listed in the pyx file, which is in a subfolder, are expected to be in paths relative to the file where the setup.py file is, or perhaps relative to the folder I'm calling python from.

Anyway, removing the line solved the issue.

Martino
  • 729
  • 6
  • 18