-1

I know this might seem like a question that has been asked over several times, though it is a bit different.

I am very new to cython, like as new as just started knowing about it since yesterday, and came to know that it can be used to convert pure python codes to binary executable, with the help of GCC.

I have a huge Python project with multiple dependencies, including Stanford NER, Spacy NER, some custom pure python dependencies, and some core python libraries. I want to convert this whole project with multiple code files to an executable but before that I want to convert it into a c type file that can be made into an executable through GCC.

Now, I know with cython embed option can convert a pure python file into a .C type file, but does that also grab the used dependencies, or does it just convert the specific .py file to .C file. Also, the custom dependencies use some other custom dependencies inside them, so when I convert the main driver code, will it also grab all the underlying dependencies?

I have already researched about Pyinstaller and Py2Exe, but that will not exactly serve my purpose, as I want a .C code that can be compiled later on.

KaranJ
  • 48
  • 1
  • 10
  • You'll notice the answer in that question is quite involved and even then doesn't deal with trying to bundle the Python standard library and all your other dependencies (some of which have their own binary modules). Don't do this. It isn't the right tool – DavidW Dec 18 '19 at 09:59

1 Answers1

0

disclaimer: Not sure I'm the best person to answer this.

I don't think cython is very suitable for what you want to do (that is automatic conversion of large python project to C). Cython sill require quite significant effort to use (you need to specify types, you often need to re-arrange the code. Sometimes it is rather difficult to specify proper types for anything more complex (e.g. numpy arrays)).

Honestly I don't have much experience in using 3rd party libaries from within cython (.pyx), since I use cython for excatlly the opposite purpose (to write low-level performance critical computational core in cython (which has no dependencies) which is than imported in high-level python scripts). I think that is the intended use case.

=>

I would follow that strategy - write only the minimum of code in cython, and than outer high-level python as a glue-language to bind the external libraries.

Prokop Hapala
  • 2,424
  • 2
  • 30
  • 59
  • I don't think this is quite right. Cython _should_ be able to compile most Python code unchanged (and actually does pretty well). It's plainly unsuited for what OP is suggesting, but not really for the reasons you suggest – DavidW Dec 18 '19 at 09:55
  • Got my answer here, https://stackoverflow.com/questions/30157363/collapse-multiple-submodules-to-one-cython-extension, but the process seems so complicated, I couldn't even wrap my head around it. What I understood though, was that I can either create a huge .py file and then convert it to C, or I can convert individual modules to SO files, Create a custom importer as mentioned in the answer above, such that when the .C file is compiled it can dynamically pick up the locations of the compiled packages. Am I right? – KaranJ Dec 18 '19 at 10:11
  • @KaranJ Your summary of the two options is about right. The real challenge is handling external dependencies. I'd really recommend PyInstaller (or similar) instead - they can bundle compiled Cython modules so you can meet your requirements (your code is compiled + big exe) – DavidW Dec 18 '19 at 10:16
  • @DavidW - yes, cython can compile python code even without type-anotations (if you mean that), but what's the purpose, when it is still as slow as python? (OK, maybe the goal here is not to gain performnce by moving from python to C, but I'm not sure what is the goal then) – Prokop Hapala Dec 18 '19 at 10:28
  • @ProkopHapala The goal here seems to be to have a self-contained executable and doesn't make any claims about speed. But yes, I agree that type-annotations are generally a good idea if you want to improve performance. – DavidW Dec 18 '19 at 10:34
  • @Prokop Hapla - The goal here was majorly to hide my Python code logic, string literals and other values did not concern me, only the logic used in the py was to be hidden. I did create a .C from one of the pure python modules, but when I looked into it, seems like a lot pf python code was retained into it. I have resorted to use .pyc files directly, however, it still doesn't solve my purpose. – KaranJ Dec 18 '19 at 10:35
  • @DavidW I would have used PyInstaller but my purpose is not to create a .exe out of the ode directly. I want to hide the source in such a way that it can be directly used to compile an executable whenever needed and hide my actual code. Also a lot of dependencies are not standard py libraries that can be installed through pip, most of them are custom made libraries basically designed to make the code modular instead of one big blob. I want a .C file that can not just hide the base code bust also include this sub libraries. – KaranJ Dec 18 '19 at 10:41
  • @KaranJ : *goal here was majorly to hide my Python code logic* ... uh, what is it good for? Some kind of obfuscation so that other people does not understand what you do? – Prokop Hapala Dec 18 '19 at 12:57
  • @Prokop Hapla - yeah something of the sort, but the existing obfuscation methods don't meet my exact needs, I don't want python code to be hidden as a python code itself. Some other sort of encryption of the code will help. I'm researching on code encryption now. – KaranJ Dec 18 '19 at 13:43