ModuleNotFound exception with zipapp and compiled C code

Question

I encounter strange behavior using Python's zipapp feature.

Python version: 3.10.5

# these steps come from the docs: https://docs.python.org/3/library/zipapp.html
$ mkdir myapp

# this is the library I'd like to use
$ echo "cysystemd==1.5.3" > requirements.txt
$ pip install -r requirements.txt --target myapp

# prepare the zipapp entry point, just import the library
$ echo "from cysystemd import journal" > myapp/__main__.py

# executing the directory works fine. the import takes place and the program terminates successfully.
$ python3 myapp/

# zipapping
$ python -m zipapp -p "python3" myapp 

# now executing again, this time the zipapp:
$ ./myapp.pyz                                                                               
Traceback (most recent call last):
  File "/home/samba/.pyenv/versions/3.10.5/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/samba/.pyenv/versions/3.10.5/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/samba/git_wsl/zipapp_experiment/./myapp.pyz/__main__.py", line 1, in <module>
  File "/home/samba/git_wsl/zipapp_experiment/./myapp.pyz/cysystemd/journal.py", line 7, in <module>
ModuleNotFoundError: No module named 'cysystemd._journal'

I can't get my head around this import issue. _journal refers to a compiled C module which is present in both locations, the myapp directory as well as the zipapp myapp.pyz:

$ find myapp/ | grep journal                                                              
myapp/cysystemd/journal.py
myapp/cysystemd/__pycache__/journal.cpython-310.pyc
myapp/cysystemd/_journal.cpython-310-x86_64-linux-gnu.so

$ unzip -l myapp.pyz| grep journal                                                    
     4880  2022-06-21 21:39   cysystemd/journal.py
   301104  2022-07-19 18:47   cysystemd/_journal.cpython-310-x86_64-linux-gnu.so
     5043  2022-07-19 18:47   cysystemd/__pycache__/journal.cpython-310.pyc

Any ideas what is wrong here? I suspect an issue related to sys.path but I have no idea how to get this fixed in a zipapp archive.

Answer 1

That's a limit imposed by Python's import mechanism. Python has various import hooks:

>>> import sys
>>> sys.path_hooks
[zipimport.zipimporter,
 <function _frozen_importlib_external.FileFinder.path_hook...]

And zipimport says:

Any files may be present in the ZIP archive, but importers are only invoked for .py and .pyc files. ZIP import of dynamic modules (.pyd, .so) is disallowed.

To get around that limitation tools in that space (like pex, shiv, ...) use the zipapp approach but they will extract the zip file before running it, transparently - hence circumventing the limitation of the zipimport module.

Please have a look at using cysystemd with the shiv tool here:

• https://gist.github.com/miku/e0a3d4abb2ba7dbb9ca5f11369fc64f4

---

PS. There's quite a bit of a packaging options; Python Packaging Authority has more info on that; some talks on the topic: "The Packaging Gradient", PyBay2017, "Packaging Python Application", PyCon Balkan 2018, ...