-1

I have a class named neuron in Cython syntax which works perfectly fine with Jupyter inline using magic (%%cython):

cdef class neuron: pass

and I am trying to cythonize this so that I can import it on a cluster and run larger scale experiments using Jupyter on a conda environment. My setup.py file looks like this:

from setuptools import setup, Extension
from Cython.Build import cythonize

extensions = [
Extension("Neuronal_Cascades_cython_Base1", ["Neuronal_Cascades_cython_Base1.pyx"]),
]
setup(
    name="Neuronal_Cascades_cython_Base",
    ext_modules=cythonize(extensions),
)

Cythonize works fine and .so and .c files created fine without any errors. But when I'm importing these two modules in Jupyter notebook, I get the import error:

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-3-639e2d302e82> in <module>
      1 import matplotlib.pyplot as plt
      2 import numpy as np
----> 3 from Neuronal_Cascades_cython_Base1 import neuron
      4 import os
      5 import pickle

ImportError: cannot import name 'neuron' from 'Neuronal_Cascades_cython_Base1'     (/Users/bengieru/Neuronal_Cascades/Cython/Neuronal_Cascades_cython_Base1.cpython-37m-darwin.so)

Can anyone tell what I am doing wrong? I feel like it may be related with the setup.py importing dependencies but I'm not sure how to fix it.

Ulgen
  • 93
  • 1
  • 8
  • `cimport `numpy` would normally require you to set the path to the Numpy headers in setup.py, but that should give an obvious error message and not generate the .so files. Have you tried importing the modules as `import Neutronal_Cascades_cython_Base1` and using `dir` to inspect what is available? – DavidW Dec 18 '20 at 09:36
  • When I do `dir` to the `import Neutronal_Cascades_cython_Base1` I got `['__builtins__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', '__test__']`. Not sure what this means exactly. Do you think it is my syntax? @DavidW – Ulgen Dec 18 '20 at 14:51
  • I thought `Neuronal_Cascadescython_Base2` might be related because importing either of the modules separately gives me the same error and since the dependencies are different for both modules, I thought it may be relevant for reproduction. @ead – Ulgen Dec 18 '20 at 14:54
  • Might be worth printing `Neutronal_Cascades_cython_Base1.__file__` and checking that it is actually the generated .so file that you think it is. – DavidW Dec 18 '20 at 14:58
  • Yes indeed it is `'/Users/bengieru/Neuronal_Cascades/Cython/Neuronal_Cascades_cython_Base1.cpython-37m-darwin.so'` @DavidW – Ulgen Dec 18 '20 at 15:00
  • I'd also try running `setup.py` with a `--force` argument. It's tries to look at timestamps to avoid recompiling where it isn't needed, but I have occasionally known that to go wrong. – DavidW Dec 18 '20 at 15:21
  • @ead okay I reduced it to the minimal reproducible example-- it returns the same error with `cdef class neuron: pass` – Ulgen Dec 18 '20 at 15:28
  • @DavidW adding `--force` didn't help also.. This is so annoying! – Ulgen Dec 18 '20 at 15:31
  • Given the output of dir, there is nothing in your module. Maybe setup.py picks up wrong pyx-file. – ead Dec 18 '20 at 15:34
  • It does `import Neuronal_Cascades_cython_Base1` correctly though. There is just nothing in it. How do I possibly fix it? @ead – Ulgen Dec 18 '20 at 15:40
  • The pyx you claim to be the source of your module, isn’t the source of the module. Why this is the case cannot be deducted from your question (typo, another mix up) – ead Dec 18 '20 at 16:08
  • @ead what do you mean? `neuron` is the class in `Neuronal_Cascades_cython_Base1.pyx` – Ulgen Dec 18 '20 at 16:17
  • I think the worry is that somehow the wrong file is being complied (since otherwise `neuron` really should appear). We don't know which wrong file or what exactly is happening but there's no other obvious explanation. It might be worth trying with a fresh file with a completely different name and seeing if that works. – DavidW Dec 19 '20 at 16:01
  • If you try to use the module in the Jupiter notebook, it will not work if another version of the module was already used/loaded. You will have to restart it. Simply reimporting a c-extension will not work: https://stackoverflow.com/a/55172547/5769463 – ead Dec 24 '20 at 06:50
  • @ead I am not trying to reimport though. This doesn't make any sense :/ I tried doing it on a different device and I got the same error. I changed the name of `Neuronal_Cascades_cython_Base1.ipynb` file that I saved as `Neuronal_Cascades_cython_Base1.pyx` in case it was trying to import `.ipynb` but still not working.. Can it possibly have to do with saving that `.ipynb` as `.pyx` and changing ` "execution_count": null` to ` "execution_count": None` in the `.pyx`? If I don't do this, setup.py throws an error.. – Ulgen Dec 24 '20 at 15:13

1 Answers1

1

I found my mistake. After trying a million things, I figured it has nothing to do with my setup.py or my source code. There were three issues:

  1. setup.py needs to be in the parent directory to work properly.
  2. an empty __init__.py file needs to be in the child directory.

Parent

|--Neuronal_Cascades_Base1

| |--neuron.pyx

| |--init.py

|--setup.py

(Can someone help me formatting this tree nicely?)

  1. When you save a .ipynb file using 'save as' on the Jupiter notebook drop down menu as neuron.pyx creates a file that look weird and all and there are bunch of other unnecessary metadata. So, deleting everything inside neuron.pyx and simply copy-pasting the original cython source code in the neuron.pyx resolved the issue.
Ulgen
  • 93
  • 1
  • 8
  • Similar error in my code, adding a __init__.py along the cython .pyx fixed it. Thank you for sharing your answer !! – erwanp Jun 12 '21 at 20:53