0

I have an extension that I compile with cython into the related *.pyd or *.so files, which are then imported into my very basic main.py:

if __name__ == "__main__":
    from my_extension import Main
    Main()

The issue is that I am trying to compile and package all the required modules into this single extension for use with pyinstaller.

By default, pyinstaller will look for any imports listed in main.py, but it'll only see that it needs my_extension.Main and will not look for what my_extension.Main needs. This leads to a situation where I need to include those external modules as hiddenimports for pyinstaller, which works fine, but I want to gain the advantage of having the external modules compiled as well into the singular extension, or otherwise into their own extensions.

Is there an easy way to tell cython to include those external modules in it's compilation?

UPDATE

So I ended up finding a solution, but it doesn't exactly work. Here are the steps I followed:

  1. Create a virtual environment in the working directory with python -m venv .venv and enter it
  2. Install related dependencies with:
python -m pip install -U pip wheel cython pyinstaller
python -m pip install -r requirements.txt
  1. Run the following setup.py file using python setup.py build_ext:
from setuptools import Extension, setup
from Cython.Build import cythonize
from pathlib import Path

extensions = []

for directory in Path(".venv/Lib/site-packages").iterdir():
    if directory.is_dir() and ("cython" not in str(directory.name).lower() and "pyinstaller" not in str(directory.name).lower()):
            items = list(directory.glob("*[!__init__].py"))
            for i in list(directory.glob("*[!__init__].pxd")):
                items.append(i)
            if len(items) > 0:
                for i in range(len(items)):
                    items[i] = str(items[i])
                extensions.append(
                    Extension(
                        str(directory.name),
                        items,
                    )
                )

setup(
    ext_modules=cythonize(
        extensions,
        language_level=3,
        exclude_failures=True,
    ),
)

The problem comes in that last step - although the setup.py is able to find any related *.py files listed in the module's directory, it will often fail with certain libraries entirely, and the option exclude_failures=True doesn't bypass this. One example of a problematic library is numpy, which I could just explicitly remove from the list of modules to compile if it's found, but I'm trying to write this to be programmatically done instead of having any hard-coded sections.

I'm inclined to believe that this is a wasted effort, and that I could simply get along by compiling my source code, then packaging it alongside the dependencies into a single executable with pyinstaller, but if someone out there has any ideas then I'm all ears.

Hunter_AP
  • 53
  • 1
  • 6
  • Pyinstaller is designed for this (although you may have to give it some help finding the modules used by Cython). Cython really isn't. You can make it work due simple cases but it's pretty complicated to do anything beyond that – DavidW May 07 '22 at 07:44
  • Cython also doesn't have a way of *automatically* identifying these external modules. – DavidW May 07 '22 at 07:51
  • I'm using pyinstaller to package the python runtime, my code, and the libraries needed into a single executable, which works fine so far. But I was hoping to find a way to programmatically identify what libraries get imported (especially if I'm using a virtual environment located in my working directory that contains those imported modules). I did a brutish method of adding the modules inside the `site-packages` dir to the cython setup.py, but I get some errors that I don't quite understand (like the PIL module's `__init__.py` having `from . import _version` causing a cython error) – Hunter_AP May 08 '22 at 13:42
  • My question was closed saying that it's a duplicate of this one: https://stackoverflow.com/questions/30157363/collapse-multiple-submodules-to-one-cython-extension But that doesn't really answer my question - that one asks how to compile multiple user-made files into one file, whereas I'm asking about compiling my one project file and it's dependencies. It's a small enough distinction that the suggested solution in the other question doesn't work here. – Hunter_AP May 08 '22 at 13:46
  • I've reopened it but honest I think it's still a duplicate. The upshot is the way that's proposed there is viable for the simplest possible case. For **anything** more complicated you just can't do this. This is a dead-end. You will not get it to work – DavidW May 08 '22 at 16:25

0 Answers0