2

and thanks in advance for whatever help you can give me on this problem.

I am working to deploy a python project with a cython-generated .c file with the following configuration for the setup.py file:

. . .

setup(

    . . .,

    packages=[
        "hatchet",
        "hatchet.readers",
        "hatchet.util",
        "hatchet.external",
        "hatchet.tests",
        "hatchet.cython_modules.libs",
    ],
    install_requires=["pydot", "PyYAML", "matplotlib", "numpy", "pandas"],
    ext_modules=[
        Extension(
            "hatchet.cython_modules.libs.subtract_metrics",
            ["hatchet/cython_modules/subtract_metrics.c"]
        ),
        Extension(
            "hatchet.cython_modules.libs.graphframe_modules",
            ["hatchet/cython_modules/graphframe_modules.c"]
        )
    ],
)

In our main implementation directory, the __init__.py file contains the following lines of code:

from .graphframe import GraphFrame
from .query_matcher import QueryMatcher

__version_info__ = ("1", "2", "0")
__version__ = ".".join(__version_info__)

When building this project using python setup.py install I get the following error:

Traceback (most recent call last):
File "/usr/WS1/scullyal/hatchet/hatchet/graphframe.py", line 24, in <module>
    import hatchet.cython_modules.libs.graphframe_modules as _gfm_cy
ModuleNotFoundError: No module named 'hatchet.cython_modules.libs.graphframe_modules'
Traceback (most recent call last):
  File "setup.py", line 10, in <module>
    from hatchet import __version__
  File "/usr/WS1/scullyal/hatchet/hatchet/__init__.py", line 9, in <module>
    from .graphframe import GraphFrame
  File "/usr/WS1/scullyal/hatchet/hatchet/graphframe.py", line 24, in <module>
    import hatchet.cython_modules.libs.graphframe_modules as _gfm_cy
ModuleNotFoundError: No module named 'hatchet.cython_modules.libs.graphframe_modules'

The problem, as I have identified it, comes from a psuedo-cyclic dependency where the setup.py script must run the __init__.py to perform its install; however this line in __init__.py -- from .graphframe import GraphFrame -- is dependent on a extension which has not been built yet and will not be built until the install process gets to the building extensions phase.

We require this line for namespace reasons (see the top answer here for an explanation on importing modules in init.py module imports and __init__.py in Python) to ensure that the user facing API we are developing does not become too and verbose. I have done some research, but I was unable to find a way to specify the order in which the install script executes. I also would like to avoid a hacky solution which attempts to build the module when this exception is encountered.

Does anyone have a possible solution to this problem? A way that I can keep the import line in the __init__.py and ensure that the necessary .c file is built before executing it?

cscully-allison
  • 21
  • 2
  • I may be missing something, but I don't see why setup.py would need to import `__init__.py` - that just seems odd? – DavidW Aug 02 '20 at 08:34
  • Hey David, I agree it does seem odd to me too, but it always does it as part of the build process. If I could defer the running of the `__init__.py` that would be ideal. – cscully-allison Aug 03 '20 at 15:33
  • Is it that you have a partly/unsuccesfully/old installed version somewhere and it's importing that? Can you delete that existing version and try again? The paths look like that might be the case. – DavidW Aug 03 '20 at 16:24
  • Thank you for the suggestion. I have now checked in my virtual-environment site packages and unfortunately I do not have any partially installed 'hatchet' package there. As for the paths, I'm working on a server with workspaces allocated to each user and that's where the `/usr/WS1/scullyal/` directory comes from; if that's what you are referring to. If you are referring to the module paths, I have added this 'hatchet' directory to my PYTHONPATH manually so that I can access it as though it were installed in my site-packages. – cscully-allison Aug 03 '20 at 22:42
  • I'm afraid I'm a bit stumped then - sorry. One final question: setup.py line 10 (which is mentioned in the traceback). Could you make sure this is actually in the setup.py you show. – DavidW Aug 04 '20 at 06:04
  • I checked it out, and that line is in there. And I think you solved my problem: I feel stupid for being so unobservant, but it was that import statement which was running the init.py and triggering this cyclic dependency. As a solution, our group is moving the version info out of the `__init__.py` to a dedicated `version.py` file. Thank you for your help with this. – cscully-allison Aug 04 '20 at 21:44

0 Answers0