Embedded python in C : Is there a way to import numpy properly from zipped python archive?

Question

Seems like I can not import numpy properly from within C application which embeds python, when using local .zip python file containing all the python files and binaries rather then standard python installation. This is the error I'm getting :

zipimport: zlib available
python27.zip\Lib\site-packages\numpy_globals.pyc has bad mtime
zipimport: zlib available
import numpy._globals # loaded from Zip python27.zip\Lib\site-packages\numpy_globals.py
zipimport: zlib available
python27.zip\Lib\site-packages\numpy__config__.pyc has bad mtime
zipimport: zlib available
import numpy.config # loaded from Zip python27.zip\Lib\site-packages\numpy__config__.py
zipimport: zlib available
python27.zip\Lib\site-packages\numpy\version.pyc has bad mtime
zipimport: zlib available
import numpy.version # loaded from Zip python27.zip\Lib\site-packages\numpy\version.py
zipimport: zlib available
python27.zip\Lib\site-packages\numpy_import_tools.pyc has bad mtime
zipimport: zlib available import numpy._import_tools # loaded from Zip python27.zip\Lib\site-packages\numpy_import_tools.py
zipimport: zlib available
python27.zip\Lib\site-packages\numpy\add_newdocs.pyc has bad mtime
zipimport: zlib available
zipimport: zlib available
import math # builtin
zipimport: zlib available
import numpy.lib.info # loaded from Zip python27.zip\Lib\site-packages\numpy\lib\info.pyc
zipimport: zlib available
zipimport: zlib available
zipimport: zlib available
python27.zip\Lib\site-packages\numpy\core\info.pyc has bad mtime
zipimport: zlib available
import numpy.core.info # loaded from Zip python27.zip\Lib\site-packages\numpy\core\info.py
import numpy.core # loaded from Zip python27.zip\Lib\site-packages\numpy\core__init__.pyc
import numpy.lib.type_check # loaded from Zip python27.zip\Lib\site-packages\numpy\lib\type_check.pyc
import numpy.lib # loaded from Zip python27.zip\Lib\site-packages\numpy\lib__init__.pyc
import numpy.add_newdocs # loaded from Zip python27.zip\Lib\site-packages\numpy\add_newdocs.py
import numpy # loaded from Zip python27.zip\Lib\site-packages\numpy__init__.py
cannot import name multiarray

But when I extract abovementioned .zip file, and run python.exe interpreter and import numpy, everything works fine.

I've build python 2.7.13 from source, x86 Release. After that installed numpy-1.11.3+mkl-cp27-cp27m-win32.whl file from here, then made a python .zip archive with all neccessary files following common folder structure.

This is how my C code looks like :

int main(int argc, char **argv)
{
    Py_VerboseFlag++;
    Py_NoSiteFlag++;

    Py_SetProgramName(argv[0]);
    Py_SetPythonHome(".");

    Py_Initialize();

    PyRun_SimpleString("import sys");
    PyRun_SimpleString("sys.path = ['.','python27.zip','python27.zip/DLLs','python27.zip/Lib', 'python27.zip/Lib/site-packages']");;
    PyRun_SimpleString("print sys.path");

    printf("\n");

    char filename[_MAX_PATH];
    _fullpath(filename, "mod1.py", _MAX_PATH);

    PyObject* main_module = PyImport_AddModule("__main__");
    PyObject* main_dict = PyModule_GetDict(main_module);
    PyObject *pyFileObj = PyFile_FromString(filename, "r");

    if (pyFileObj == NULL) {
        return -1;
    }

    FILE *pFile = PyFile_AsFile(pyFileObj);

    if (pFile == NULL)
        return -1;

    PyObject *result = PyRun_File(pFile, filename, Py_file_input, main_dict, main_dict);

    if (!result)
        print_error();

    printf("\n\n");


    Py_Finalize();
    getchar();

    return 0;
}

module mod1.py, I'm executing contains this :

import time
import json
import numpy

if __name__ == "__main__":
    print 'Success'

I'm aware that that error comes from the fact that python can not load multiarray.pyd which is shared library.Do I need to handle this scenario separately when it comes to import?
Is it possible to import numpy properly from zipped, non-standard python installation from C application?

EDIT : Forgot to mention following :
- My OS is Windows 10 x64 Version 1511 (OS Build 10586.545)
- When I extract .zip archive in my Release folder where my C app binary resides, everything works fine, numpy is loaded properly with following sys.path

PyRun_SimpleString("import sys");
PyRun_SimpleString("sys.path = ['.', 'DLLs', 'Lib', 'Lib\\site-packages', 'Sources']");
PyRun_SimpleString("print sys.path");

EDIT 2: Also, thought to check exact version of python27.dll the multiarray.pyd is linked to (according to other posts about similar problems) and it turned out it is different then mine - it's 2.7.8. No problem, extracted specific python27.dll v2.7.8 binary from the .msi package and replaced mine which was 2.7.13, but still no luck. It is definitely something about zipimport module.

You cannot load a shared library from a zip file. This is a limitation of the operating system and applies to Windows, Linux and Mac OS. It is not an issue with Python or its zip file importer. Since Numpy contains C extensions you cannot bundle it in a zip executable. — JacaByte, Feb 18 '18 at 23:23
Thanks for the answer, I've already learned that the hard way.However, I think I made it (on Win) by utilizing MemoryModule lib written by Joachim Bauch by simply decompressing archive into memory and load stuff from there.Sample code here : https://github.com/Civa/continuum/blob/master/src/continuum/runtime/loader.c (WARNING : poorly written library with messy code since I'm still learning C) — civa, Feb 19 '18 at 19:56
Interesting, I ended up doing something similar for Linux; conceptually it's a wrapper that copies the binary into a folder in /tmp, adds the folder to ``sys.path`` and evals ``from mymodule import *`` to load the extension. I didn't know about this loader. — JacaByte, Feb 20 '18 at 03:33
Yeah, it's an amazing idea.Many thanks to author Joachim Bauch! :) I've been thinking about the same thing - to unpack stuff to disk, to some tmp location and load stuff from there, but I was just dying of curiosity - there must be some way of loading binaries from the memory :) — civa, Feb 20 '18 at 08:51
@Civa, I clicked on your link and got a 404 error. Do you mind checking to see if that link is out of date? I'd like to see it. Thanks. — JasonArg123, Jun 04 '20 at 04:15
ah sorry @JasonArg123, I've migrated all the stuff from github and might have deleted obsolete repos in the process. As soon as I find the source I will upload it to the gitlab — civa, Jun 04 '20 at 10:00
@JasonArg123 [here](https://gitlab.com/old-stuff2/continuum) is the old repo I found. If you want to see how loader works, just search for `loader.c`. Note that, this code is VERY messy. Since this is written for windows I can't be of much help since I swithced to LInux long time ago, sorry. Please feel free to do anything you want with the code as there is no license attached. Hope you will find it useful. Cheers, civa — civa, Feb 17 '21 at 12:24

Alex W · Answer 1 · 2021-03-31T16:41:44.897

The best way is probably to unzip it for now. The zipimport module is what Python uses to load modules from zip files and unfortunately it disallows the import of dynamic compiled code (probably due to security concerns, consistent with PEP 273):

Any files may be present in the zip archive, but only files *.py and .py[co] are available for import. Zip import of dynamic modules (.pyd, *.so) is disallowed.

Because the wheels provided will be platform wheels, you will also need to remove the platform extension from the .so filenames which are added by Cython as part of adherence to PEP 3149. I.e., foo.cpython-XYm.so would need to be renamed to foo.so. (Since your question asks about Python 2, this does not apply but it does going forward for Python 3)

Lastly, the unzipped folders will need to be added to the PYTHONPATH if they are not already on it.

thanks for the reply, but I abandoned that project long time ago. — civa, Apr 01 '21 at 21:38
No problem, had this issue on Spark cluster so I did an in-depth investigation and wanted others to not have to — Alex W, Apr 02 '21 at 14:23

Embedded python in C : Is there a way to import numpy properly from zipped python archive?

1 Answers1