1

With a few hours worth of effort I've finally managed to convert my Python code to C, and then that to an exe file, to find the imports aren't working. I'm wondering if anyone knows how to make it include everything? None of my searches are coming up with anything.

I'm not sure if it'll make a difference, but this is how I'm doing it:

The --inplace option wasn't working with a setup.py (as in it would generate a C file which wouldn't compile), so I'm doing it through the command line with cython --embed -o main.c main.py.

I copied the libs and include folder from Python into another one and from there I'm compiling the C files. Here's the error when I try run my script:

Traceback (most recent call last):
  File "start_tracking.py", line 5, in init start_tracking (start_tracking.c:1133)
ImportError: No module named core.constants

How would I go about including all the imports in the generated C code (without just copying everything into one file)?

DavidW
  • 29,336
  • 6
  • 55
  • 86
Peter
  • 3,186
  • 3
  • 26
  • 59
  • I've never figured out a way to do it - you probably have to manually copy all dependencies into subfolders from the site-packages directory exactly as they rest in site-packages, e.g. subfolders below where your exe is located. I believe embed just embeds the Python interpreter. – Matt Jun 24 '17 at 15:44
  • To clarify from your error you need at least `core` package installed in its entirety below the exe path. – Matt Jun 25 '17 at 04:09
  • I found another huge problem with it, I was assuming it was able to write in the actual imports, but it turns out even as an exe file it was looking up the Python folder. Only found that out after I spent a few hours writing a script to merge everything into one page haha. – Peter Jun 25 '17 at 10:19

2 Answers2

2

{You may want to use cx_freeze https://www.google.com/url?sa=t&source=web&rct=j&url=https://anthony-tuininga.github.io/cx_Freeze/&ved=0ahUKEwjxuufa7uDUAhUR0GMKHW9YCJoQFgggMAA&usg=AFQjCNExxKVSPXU3tHdi6ahNqoekGMC1Eg with wheels here, seems to do much or quite a bit of what you're asking https://pypi.python.org/pypi/cx_Freeze}

Okay this isn't exactly answering your question but it should work (albeit not the approach you were taking). I'm going to give you a huge download link for this purpose where you can just drop in a (v3.5) .py file and go. In your case a .py file would call your compiled Cython PYD (if you are in fact including your own Cython modules) through an import statement, followed by whatever other code you're calling. I think having an entire exe with all imports inside would be great but so many packages require DLLs or PYD files to function, so your build is going to always have at least an exe, the Python interpreter and all its dependencies (such as Python3x.DLL). you can try: memimporter and zipextimporter which are able to load .pyd files from memory/zip-archives without unpacking them to files, but that would just allow you to put all your extension modules in a zip file for distribution (and potentially slow the PYD & DLL loading speed required but I haven't seen benchmarks around the methods).

If you do happen to get the above working in 1 zip package for extension modules please give me your code changes!!! This already includes NumPy, SciPy, Pandas, and built against Intel MKL. Reference link for usage: https://stackoverflow.com/a/44610044/6037118 ideally I would extend the approach from that link with some encryption algo on the .py files which would get decrypted on runtime. The direct download is here but you should see my other link to clarify usage: https://www.dropbox.com/sh/2smbgen2i9ilf2e/AADI8A3pCAFU-EqNLTbOiUwJa?dl=0 I should note that your compiled PYD and its dependent DLLs should also be in the extension_modules in a subfolder compiled with Visual Studio 2015 x64 for Windows builds. That link won't run on Linux as is.

Matt
  • 2,602
  • 13
  • 36
  • Thanks, it looked promising but only flashes up for less than a second (and I moved the required files into the extension folder). I built the script in Python 2 but attempted making it compatible with 3, so I'm not sure if it's that or not. Cheers anyway :P – Peter Jun 25 '17 at 22:58
  • Try opening a cmd prompt so you can see any missing packages – Matt Jun 26 '17 at 05:55
  • Would have done that had I found how to pass in a file (can't seem to search for the right question). Dumb question haha, but how do I pass in my python file via the command line? – Peter Jun 26 '17 at 11:07
  • Should just be: embedpython scriptname functionname parameter1 parameter2 etc. No .py on the function. You probably have to write a small wrapper around your Cython module. – Matt Jun 26 '17 at 15:20
  • @Peter You also can't use the exe you'll need to compile it as a PYD then write a Py file that calls it as outlined above. `from mylibray import my function` then do whatever you want with your program. – Matt Jun 26 '17 at 16:59
  • Try the eig.py included to ensure it outputs a file when called. Then you'll know if the system is setup right on your PC. I just zipped up the whole build directory and could have forgotten to pull all the DLLs in there. I won't be at my PC for a week but if that is the case I have a program I've distributed to another PC that worked without Python installed I can zip that up then. – Matt Jun 26 '17 at 17:06
  • I have two parts to my script, and with a few minor tweaks for Python 3 I got the exe reading one of them fine. The other however, it's not liking the multiprocessing module, but I don't think that's down to your script as I also had the same problem with cython itself. If I'm understanding correctly, your script allows anyone to run .py files without needing to have external modules installed, but you still need Python itself? – Peter Jun 26 '17 at 17:54
  • So now you have to copy the entire multiprocessing folder from site-packages (usually Python/lib/site-packages into the `extension_packages` folder which lies under where your embedpython.exe is located – Matt Jun 26 '17 at 19:44
  • But no, you NEED every library imported by your program under site-packages installed under extension modules. This won't put external libraries in except NumPy, SciPy, and Pandas – Matt Jun 26 '17 at 19:55
  • @peter note my edited answer - you have to use MSVC 14 to compile it (Community VS 2015 works) – Matt Jun 27 '17 at 00:30
  • Thanks, I just tried copying the multiprocessing folder, and then everything else contained in site packages just to make sure nothing is missing, but it still gives the ImportError. I can replicate it with the following code: `from multiprocessing import Queue` `def test(): Queue()`. Also apologies as I'm entirely new to the world of compiling, but why does the py file have to be compiled to pyd if it works fine straight away? – Peter Jun 27 '17 at 08:41
  • @Peter if you are using my build it should be in extension_modules / multiprocessing / – Matt Jun 27 '17 at 08:46
  • Yeah it's copied to there and not working, though like I said, I think multiprocessing is generally not very compatible with other scripts – Peter Jun 27 '17 at 08:50
  • You have to be missing dependent packages unfortunately. You will have to figure out the dependencies of each import function and include them in extension_packages – Matt Jun 27 '17 at 08:53
  • I'm not sure about that, I've tried copying in every single file from `DLLs`, `libs` and `site-packages` to make sure I'm not missing anything that multiprocessing might use, and the script I'm running isn't using any other imports. Multiprocessing couldn't have any dependencies outside of the python folder right? I'll give your answer a vote anyway but I don't think it's gonna work for my case aha :P – Peter Jun 27 '17 at 09:04
  • Although if you copied everything over into extension_modules then it had to be something else wrong. Besides, 1 very good reason to use Cython is OpenMP and prange loops. That would be better in this code than multiprocessing – Matt Jun 27 '17 at 09:17
  • I looked up the multiprocessing cython error before (when my compiled code was acting strange) and the fix was either use OpenMP or not use cython haha. I'm heavily relying on `Queue` though so can't really switch, and I'm using it to run a single background process, not spawning lots of processes to work on the same thing, so the c implementations won't really improve anything in terms of speed :) – Peter Jun 27 '17 at 09:34
  • @Peter we have too many comments but let me state ALL your imported libraries MUST be from Python 3.5 x64 or they won't work!!! Likely why copying `multiprocessing` didn't work. Also yes you can just use the .py if you want and not compile anything. – Matt Jun 27 '17 at 16:15
  • I tried copying everything from the 3.6 installation, but yeah, I guess next time you end up using it, just throw in a `multiprocessing.Queue` somewhere and see if it works or not :) `cx_freeze` couldn't handle it either, but it appears `pyinstaller` can, so for now I'll just stick with that and the resulting 360mb folder :P – Peter Jun 27 '17 at 16:28
  • And it seems they just updated cxfreeze which apparently does everything discussed here! http://cx-freeze.readthedocs.io/en/latest/overview.html – Matt Jun 28 '17 at 15:25
1

I had the same issue.I followed this link to include all my modules in my root pyx file. However if that module imported another one, it ran into import errors.

After some research I used stickytape to convert to a single python file say main.py. Then

cython --embed -o main.c main.py

gcc main.c -I /usr/include/python3.5m -L /usr/lib/x86_64-linux-gnu/ -o main -lpython3.5m -lpthread -lm -lutil -ldl

And viola, I have a working executable from c that was converted through python

Arvind Sreekumar
  • 31
  • 4