Compile main Python program using Cython

Question

I have a Python2.6 program that can load Python modules compiled to .so files using Cython. I used Cython to compile the .py modules to .so files and everything works fine.

This is the setup.py file I use with Cython:

from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext

ext_modules = [
    Extension("ldap", ["ldap.pyx"]),
    Extension("checker", ["checker.pyx"]),
    Extension("finder", ["finder.pyx"]),
    Extension("utils", ["utils.pyx"]),
]

setup(
  name = 'bchecker',
  cmdclass = {'build_ext': build_ext},
  ext_modules = ext_modules
)

So I know I can compile Python modules using Cython (I guess Cython creates 'C' files from my Python files and then compiles them), but can I compile my main Python program to something I can execute on a Linux platform? If so, a Cython command line example would be appreciated. Thanks.

score 170 · Answer 1 · edited May 23 '17 at 12:18

170

Contrary to what Adam Matan and others assert, you can in fact create a single executable binary file using Cython, from a pure Python (.py) file.

Yes, Cython is intended to be used as stated - as a way of simplifying writing C/C++ extension modules for the CPython python runtime.

But, as nudzo alludes to in this comment, you can use the --embed switch at the command line prompt.

Here is an extremely simple example. I am peforming this from a Debian Sid workstation, using python3 and cython3..

Make sure you have python-dev or python3-dev packages installed beforehand.

1) Create a very simple Python program called hello.py

$ cat hello.py

print("Hello World!")

2) Use Cython to compile your python program into C...

cython3 --embed -o hello.c hello.py

3) Use GCC to compile hello.c into an executable file called hello...

gcc -Os -I /usr/include/python3.3m -o hello hello.c -lpython3.3m -lpthread -lm -lutil -ldl

4) You end up with a file called hello ...

$ file hello

hello: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.32, BuildID[sha1]=006f45195a26f1949c6ed051df9cbd4433e1ac23, not stripped

$ ldd hello
linux-vdso.so.1 (0x00007fff273fe000)
libpython3.3m.so.1.0 => /usr/lib/x86_64-linux-gnu/libpython3.3m.so.1.0 (0x00007fc61dc2c000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fc61da0f000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fc61d70b000)
libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007fc61d508000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fc61d304000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fc61cf5a000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fc61cd52000)
libexpat.so.1 => /lib/x86_64-linux-gnu/libexpat.so.1 (0x00007fc61cb28000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fc61c90f000)
/lib64/ld-linux-x86-64.so.2 (0x00007fc61e280000)

In this case, the executable is dynamically linked to Python 3.3 on my Debian system.

5) run hello...

$ ./hello

Hello World!

As you can see, using this method you can basically use Cython to convert your pure Python applications into executable, compiled object code.

I am using this method for vastly more complex applications - for example, a full blown Python/PySide/Qt application.

For different versions of Python, you tailor the gcc -I and -l switches to suit.

You can then package the executable as a distribution (.deb, etc.) file, without having to package the Python/PySide/Qt files - the advantage being that your application should still be able to run even after a distribution update to the same versions of Python, etc. on that distribution.

edited May 23 '17 at 12:18

Community

1
1

answered Feb 26 '14 at 11:50

Broken Man

2,026
1
14
18

7

It does not compiles module(s) into the executable. Is it possible to have a portable executable i.e every module which had been imported in source code would be compiled into that executable. @BrokenMan – Devesh Saini Oct 11 '14 at 15:35
1

@DeveshSaini I haven't spent the time trying to figure that bit out. The only way I've managed to get a full-blown python+Pyside application to compile as above, was to develop the program in the normal fashion - lots of include statements + testing with the interpreter. Then when all is ready I manually paste in all the included code and comment out the relevent include statements they replaced. Yes, it's undesirable, but like I said at first I've not spent the time to see how I can compile just the main routine and link the included modules together. – Broken Man Oct 12 '14 at 18:46
@BrokenMan What is the meaning of this error when I try to compile with GCC in windows? `Python header needed to compile C extensions, please install development version of Python` – the_prole Nov 26 '14 at 17:59
@the_prole Looks like you haven't yet installed the python-dev package (or python3-dev). You need that/those. I've added in a note about that in my post above. – Broken Man Nov 27 '14 at 09:00
@BrokenMan I am running Windows – the_prole Nov 27 '14 at 09:21
@the_prole I don't do Windows development (yet) - makes for a great gaming OS though ;) - but a quick google search that I've performed on your behalf comes up with this SO topic : http://stackoverflow.com/questions/16448710/how-to-get-python-dev-for-windows which may help you out. – Broken Man Nov 27 '14 at 09:33
@the_prole Just realised I missed the bit in your original question mentioning "GCC in windows" - my apologies, not enough coffee it seems. Anyway, https://docs.python.org/2/using/windows.html might help you out, which links to https://www.python.org/downloads/windows/ - and also try googling for "windows python-dev" which might provide you with further helpful information. If you find a solution to your problem, it might be a good idea to post your solution here. Rgds. – Broken Man Nov 27 '14 at 09:46
4

What about static linking of libpython? – James Mills Mar 03 '15 at 01:11
@JamesMills This might help you : http://stackoverflow.com/questions/8366544/can-i-statically-link-cython-modules-into-an-executable-which-embeds-python?rq=1 – Broken Man Mar 25 '15 at 11:20
@Broken Man i have Anconda python3, i used your step (3) "gcc -Os -I /usr/include/python3.3m -o hello hello.c -lpython3.3m -lpthread -lm -lutil -ldl" and used the Anconda include directory in the argument but can not include Lpython, how to do that? and could you explain step3 code in detail. – user1779646 Sep 29 '16 at 17:15
@user1779646 I have never used Anaconda python so don't know how it works. Perhaps it would make for a standalone question in its own right, referring to this answer? – Broken Man Sep 29 '16 at 18:49
Can you elaorate on how to convert py to c++ file and then to a binary?..In terms of command lines? – repzero Dec 04 '16 at 16:03
@BrokenMan I have a [Cython post](http://stackoverflow.com/questions/41944883/verifying-compatibility-in-compiling-extension-types-and-using-them-with-cdef) you may be able to provide insight on. – ballade4op52 Jan 31 '17 at 20:57
@HaydenDarcy the example given is for Debian Linux. I haven't tested it in Windows. – Broken Man Nov 02 '17 at 13:32
4

For python 2.7 `cython --embed -o /tmp/a.c a.py` and `gcc -Os -I /usr/include/python2.7 -o /tmp/a/tmp/a.c -lpython2.7 -lpthread -lm -lutil -ldl` – dsaydon Feb 20 '18 at 18:49

score 20 · Answer 2 · edited May 23 '17 at 12:26

20

Take a look at the answers to Can Cython compile to an EXE? which, contrary to all the other answers here say that yes, it is possible to compile to an executable.

The links at Embedding Cython seem to be a good place to start, but it's not Cython's primary purpose so I don't know how straightforward it would be.

edited May 23 '17 at 12:26

Community

1
1

answered Feb 24 '11 at 18:00

Scott Griffiths

21,438
8
55
85

score 4 · Answer 3 · edited Dec 14 '15 at 14:02

4

I don't know if this will help or not but Nudzo is correct. You can get it with cython --embed -o main.o main.py and then I try to compile the result with cl/EHsc

edited Dec 14 '15 at 14:02

Stefan Ferstl

5,135
3
33
41

answered Dec 14 '15 at 12:58

Jake Cook

71
5

score 0 · Answer 4 · answered Oct 06 '21 at 13:47

Look at this Post:

cython <cython_file> --embed

and then just

gcc <C_file_from_cython> -I<include_directory> -L<directory_containing_libpython> -l<name_of_libpython_without_lib_on_the_front> -o <output_file_name>

here an example:

cython3 main.py --embed
gcc main.c -I /usr/include/python3.8/ -L /lib/x86_64-linux-gnu/ -l python3.8 -o main

score -27 · Answer 5 · answered Feb 24 '11 at 14:06

-27

You can't, Cython is not made to compile Python nor turn it into an executable.

To produce an .exe file, use py2exe.

To produce a package for Mac or Linux, use the regulard packaging system process as there is nothing specific about a script language program in Unix env.

answered Feb 24 '11 at 14:06

Bite code

578,959
113
301
329

3

Well it allows me to turn my .py files into .so files. See the above script I use. Don't the .so files contain executable code? – mozza Feb 24 '11 at 14:21
It's not the purpose. Cython is for compiling C/C++ Python extensions, not turn Python scripts to executable. – Bite code Feb 24 '11 at 15:01
14

Misinformation. There's `cython --embed` which can produce executables with embed Python. – nudzo Jan 04 '14 at 11:59
1

The resulting executable still depends on libpython unless you link it statically. Plus it will not produce one executable out of all your files, but one executable out of your entry point. – Bite code Jan 05 '14 at 13:00
3

@e-satis Yes, the executable will be dynamically linked to the version of libpython it was compiled against at the time. What's the issue? If it's for Linux, and if you're planning on distributing your application, you just package it to whatever linux distribution format you're targeting - just like every other binary package. e.g. myapp_1_wheezy_amd64.deb would be your compiled python app linked against either python2 or python3 for that distro. For Windows just supply the required python. Again, I fail to see the problem here, or the objection. – Broken Man Jul 09 '14 at 09:11
I would have to highlight his point here. You are *embedding* the whole python interpreter into the entry point of your executables that then is being called and processing you application code as python. That is not what you need when you compile to affect performance of your application. – iantonuk May 27 '23 at 17:34

Compile main Python program using Cython

5 Answers5

Linked