How to use cython_freeze on Windows to build a single executable file?

Question

I am running Python 3.5 on Windows 10 and I would like to compile my Python code into a single executable to share with some end users. I'm using Cython 0.25.2 to try to get this done.

I've got a HelloWorld program working, by using the Cython --embed flag. From the Windows command prompt Cython makes a .c file:

# (myVirtualEnv) > cd (pathToSourceCode)
# (myVirtualEnv) > py (pathToVirtualEnv)\Scripts\cython.exe helloWorld.pyx --embed

This gives me a HelloWorld.c file. Then I open the MSVC++ 2015 x86 Native Command Prompt and compile the HelloWorld.c file into a Windows exe:

# > cd (pathToSourceCode)
# > cl.exe  /nologo /Ox /MD /W3 /GS- /DNDEBUG -I(pathToVirtualEnv)\Include -I(pathToVirtualEnv)\PC /TchelloWorld.c /link /OUT:"helloTest.exe" /SUBSYSTEM:CONSOLE /MACHINE:X86 /LIBPATH:(pathToVirtualEnv)\Lib /LIBPATH:C:\Python35\libs

This works well. When run from the windows command prompt my new helloTest.exe program prints to the command line. I need to figure out how to get the 1st set of commands (from the regular command prompt) into a distutils setup script, which I haven't done. Perhaps I can make the windows compiler commands into a make file or batch file or something.

The shortcomings here are that I want to be able to build up many modules and packages together and I also don't want to ensure that the end user has all of the dependencies (Python 3.5 and all of the needed modules) in place.

I haven't been able to figure out how to build multiple Cython files and then link them into the final Windows .exe. However, seeing how I also want to include Python itself and any included python modules, I really need a freeze tool in addition to some pre-compilation processing.

Cython has a demo folder for cython_freeze. This looks perfect. It says it pulls all included modules, including Python itself, into one c file for compilation. The example file includes a readme and a make file for gcc. I don't fully understand what's going on here, and since I'm using MSCV cl.exe, I think the right thing to do is to find/ modify this to work for cl.exe. I lack the skills to do this. In fact, because I'm on windows I got cython from pre-compiled wheels and I'm not even sure if cython_freeze exists in my virtualenv (I did a search and didn't come up with anything). Similarly, "cython freeze" web searches also came up empty. Thus far I've got the cython --embed HelloWorld program working and I've downloaded the cython source from GitHub to review the cython_freeze readme and make files.

So, my questions:

How can I use cython_freeze and cl.exe to build a single, windows executable that contains python, my main program and included packages/modules (standard or my custom ones)?
Because I installed Cython 0.25.2 from a wheel pre-compiled for Windows, is cython_freeze even in my virtualenv already? If so, how do I use it? If not, how do I "install" it?

I'm definitely open to other non-cython_freeze ways of doing this. However, I should note that I've looked at several other options that didn't look nearly as promising:

py2exe and cx-freeze and other similar freeze tools zip up the custom code but still require Python and all modules to be installed. This put too much on my users. I really want them to be able to click.exe and (as long as they are on Windows) have it run.
I've considered using gcc as the compiler, but since MSVC 2015 is used to compile Python for windows and all of the extensions it sounds like this would open the door to lots of compatibility problems with python itself.
I'm also leaning toward cython as I have some fairly slow "big math" routines in numpy that are taking too long, so I'm motivated to learn more about cython to speed those up soon too.

In addition to windows10 and Python 3.5, I'm using pip and virtualenv to manage my packages and create my virtualenv. I'm not using ant or conda. Cython, and most my other packages, are installed from pre-compiled wheels, as linked above.

UPDATE:

The spirit of my question is actually about packaging Python for users who don't have Python installed. Matt gives an excellent way to do that (and a good education on compiling Python). Biswa_9937 and S.Moncayo correctly point out that Py2Exe (Python versions 3.4 and earlier) and PyInstaller (works on Python 3.5) can also achieve this goal. These are great answers and very helpful.

However, the details of my question are about getting Cython_freeze to work on Windows. No one has attempted that, so I'm leaving the question unanswered in hopes that a Cython guru can weigh in.

Have you tried pyinstaller>. however the code to build executables by cython_freeze requires you to create a setup.py file first. refer: https://stackoverflow.com/questions/45004621/how-to-change-py-to-exe/45005045#45005045 — Dragon, Jul 10 '17 at 06:35
@Biswa_9937: I looked at pyinstaller and I believe it is a freeze tool that just zips up source code (like py2exe and cx-freeze) and still depends on python and all dependent modules being properly installed on the host computer. If so, it doesn't meet my needs. Also, it looks like pyinstaller does not have the ability to do more than 1 file (no zip file import mode): http://python-guide-pt-br.readthedocs.io/en/latest/shipping/freezing/ — Casey, Jul 10 '17 at 06:46
pyinstaller creates exe file not needing Python to run it. It does exactly what u need. — Alex B, Aug 23 '18 at 20:06

score 1 · Accepted Answer · edited Jun 20 '20 at 09:12

I'm not going to answer your question (directly) but get you the result I think you want. Instead I'm going to give you a link to a precompiled Python 3.5 x64 Windows build embedded copy with NumPy SciPy Pandas installed and you can just download and add every library you import from Python/libs/site-packages/ into the extension_modules/ directory and plug and play only using your .py script. https://stackoverflow.com/a/44610044/6037118 you may need to scroll to my answer. Besides freeze in my understanding won't put everything into the exe anyway you'll still need subfolders with imported libs.

Here's the compiled embedded Python 3.5 x64 VS 2015 link w/ NumPy, SciPy, Pandas, Intel MKL:

https://www.dropbox.com/sh/2smbgen2i9ilf2e/AADI8A3pCAFU-EqNLTbOiUwJa?dl=0

HOW TO COMPILE YOURSELF

And rather than recompile this to x32, I'll teach you how to fish per se. Here's the Visual Studio project call_function.c file:

#include <Python.h>

int
main(int argc, char *argv[])
{
    PyObject *pName, *pModule, *pDict, *pFunc;
    PyObject *pArgs, *pValue;
    int i;

    if (argc < 3) {
        fprintf(stderr, "Usage: call pythonfile funcname [args]\n");
        return 1;
    }
    
    Py_SetPath(L"python35.zip;extension_modules//;); //all your libraries from site-packages must be installed under extension_modules directory like \extension_modules\numpy\
    Py_Initialize();
    pName = PyUnicode_DecodeFSDefault(argv[1]);
    /* Error checking of pName left out */

    pModule = PyImport_Import(pName);
    Py_DECREF(pName);

    if (pModule != NULL) {
        pFunc = PyObject_GetAttrString(pModule, argv[2]);
        /* pFunc is a new reference */

        if (pFunc && PyCallable_Check(pFunc)) {
            pArgs = PyTuple_New(argc - 3);
            for (i = 0; i < argc - 3; ++i) {
                pValue = PyLong_FromLong(atoi(argv[i + 3]));
                if (!pValue) {
                    Py_DECREF(pArgs);
                    Py_DECREF(pModule);
                    fprintf(stderr, "Cannot convert argument\n");
                    return 1;
                }
                /* pValue reference stolen here: */
                PyTuple_SetItem(pArgs, i, pValue);
            }
            pValue = PyObject_CallObject(pFunc, pArgs);
            Py_DECREF(pArgs);
            if (pValue != NULL) {
                printf("Result of call: %ld\n", PyLong_AsLong(pValue));
                Py_DECREF(pValue);
            }
            else {
                Py_DECREF(pFunc);
                Py_DECREF(pModule);
                PyErr_Print();
                fprintf(stderr, "Call failed\n");
                return 1;
            }
        }
        else {
            if (PyErr_Occurred())
                PyErr_Print();
            fprintf(stderr, "Cannot find function \"%s\"\n", argv[2]);
        }
        Py_XDECREF(pFunc);
        Py_DECREF(pModule);
    }
    else {
        PyErr_Print();
        fprintf(stderr, "Failed to load \"%s\"\n", argv[1]);
        return 1;
    }
    Py_Finalize();
    return 0;
}

And here's the Visual Studio solution file you can save out as embedpython.sln:

Microsoft Visual Studio Solution File, Format Version 12.00
# Visual Studio 14
VisualStudioVersion = 14.0.24720.0
MinimumVisualStudioVersion = 10.0.40219.1
Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "embedpython", "embedpython.vcxproj", "{51062161-3FE4-42F2-9B89-BEB15E8F7590}"
EndProject
Global
    GlobalSection(SolutionConfigurationPlatforms) = preSolution
        Debug|x64 = Debug|x64
        Debug|x86 = Debug|x86
        Release|x64 = Release|x64
        Release|x86 = Release|x86
    EndGlobalSection
    GlobalSection(ProjectConfigurationPlatforms) = postSolution
        {51062161-3FE4-42F2-9B89-BEB15E8F7590}.Debug|x64.ActiveCfg = Debug|x64
        {51062161-3FE4-42F2-9B89-BEB15E8F7590}.Debug|x64.Build.0 = Debug|x64
        {51062161-3FE4-42F2-9B89-BEB15E8F7590}.Debug|x86.ActiveCfg = Debug|Win32
        {51062161-3FE4-42F2-9B89-BEB15E8F7590}.Debug|x86.Build.0 = Debug|Win32
        {51062161-3FE4-42F2-9B89-BEB15E8F7590}.Release|x64.ActiveCfg = Release|x64
        {51062161-3FE4-42F2-9B89-BEB15E8F7590}.Release|x64.Build.0 = Release|x64
        {51062161-3FE4-42F2-9B89-BEB15E8F7590}.Release|x86.ActiveCfg = Release|Win32
        {51062161-3FE4-42F2-9B89-BEB15E8F7590}.Release|x86.Build.0 = Release|Win32
    EndGlobalSection
    GlobalSection(SolutionProperties) = preSolution
        HideSolutionNode = FALSE
    EndGlobalSection
EndGlobal

Now you'll have to download the embeddable zip file (32bit one is here): https://www.python.org/ftp/python/3.5.1/python-3.5.1-embed-win32.zip and extract and place it in the directory with the built EXE file. Also, you don't need python.exe or wpython.exe it comes with, and copy all the PYD files into the \extension_modules\ directory. Note you also need to add any linked libraries .lib files in the linker:

Here's the embedpython.vsxproj file:

<?xml version="1.0" encoding="utf-8"?>
<Project DefaultTargets="Build" ToolsVersion="14.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <ItemGroup Label="ProjectConfigurations">
    <ProjectConfiguration Include="Debug|Win32">
      <Configuration>Debug</Configuration>
      <Platform>Win32</Platform>
    </ProjectConfiguration>
    <ProjectConfiguration Include="Release|Win32">
      <Configuration>Release</Configuration>
      <Platform>Win32</Platform>
    </ProjectConfiguration>
    <ProjectConfiguration Include="Debug|x64">
      <Configuration>Debug</Configuration>
      <Platform>x64</Platform>
    </ProjectConfiguration>
    <ProjectConfiguration Include="Release|x64">
      <Configuration>Release</Configuration>
      <Platform>x64</Platform>
    </ProjectConfiguration>
  </ItemGroup>
  <PropertyGroup Label="Globals">
    <ProjectGuid>{51062161-3FE4-42F2-9B89-BEB15E8F7590}</ProjectGuid>
    <Keyword>Win32Proj</Keyword>
  </PropertyGroup>
  <Import Project="$(VCTargetsPath)\Microsoft.Cpp.Default.props" />
  <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'" Label="Configuration">
    <ConfigurationType>Application</ConfigurationType>
    <UseDebugLibraries>true</UseDebugLibraries>
    <PlatformToolset>v140</PlatformToolset>
  </PropertyGroup>
  <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|Win32'" Label="Configuration">
    <ConfigurationType>Application</ConfigurationType>
    <UseDebugLibraries>false</UseDebugLibraries>
    <PlatformToolset>v140</PlatformToolset>
  </PropertyGroup>
  <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|x64'" Label="Configuration">
    <ConfigurationType>Application</ConfigurationType>
    <UseDebugLibraries>true</UseDebugLibraries>
    <PlatformToolset>v140</PlatformToolset>
  </PropertyGroup>
  <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'" Label="Configuration">
    <ConfigurationType>Application</ConfigurationType>
    <UseDebugLibraries>false</UseDebugLibraries>
    <PlatformToolset>v140</PlatformToolset>
  </PropertyGroup>
  <Import Project="$(VCTargetsPath)\Microsoft.Cpp.props" />
  <ImportGroup Label="ExtensionSettings">
  </ImportGroup>
  <ImportGroup Label="Shared">
  </ImportGroup>
  <ImportGroup Label="PropertySheets" Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'">
    <Import Project="$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props" Condition="exists('$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props')" Label="LocalAppDataPlatform" />
  </ImportGroup>
  <ImportGroup Label="PropertySheets" Condition="'$(Configuration)|$(Platform)'=='Release|Win32'">
    <Import Project="$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props" Condition="exists('$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props')" Label="LocalAppDataPlatform" />
  </ImportGroup>
  <ImportGroup Label="PropertySheets" Condition="'$(Configuration)|$(Platform)'=='Debug|x64'">
    <Import Project="$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props" Condition="exists('$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props')" Label="LocalAppDataPlatform" />
  </ImportGroup>
  <ImportGroup Label="PropertySheets" Condition="'$(Configuration)|$(Platform)'=='Release|x64'">
    <Import Project="$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props" Condition="exists('$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props')" Label="LocalAppDataPlatform" />
  </ImportGroup>
  <PropertyGroup Label="UserMacros" />
  <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'">
    <LinkIncremental>true</LinkIncremental>
  </PropertyGroup>
  <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|Win32'">
    <LinkIncremental>true</LinkIncremental>
  </PropertyGroup>
  <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'">
    <IncludePath>$(VC_IncludePath);$(WindowsSDK_IncludePath);C:\Python3;C:\Python3\include;</IncludePath>## Heading ##
    <LibraryPath>$(VC_LibraryPath_x64);$(WindowsSDK_LibraryPath_x64);$(NETFXKitsDir)Lib\um\x64;</LibraryPath>
  </PropertyGroup>
  <ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'">
    <ClCompile>
      <PreprocessorDefinitions>WIN32;_DEBUG;_WINDOWS;%(PreprocessorDefinitions)</PreprocessorDefinitions>
      <RuntimeLibrary>MultiThreadedDebugDLL</RuntimeLibrary>
      <WarningLevel>Level3</WarningLevel>
      <DebugInformationFormat>ProgramDatabase</DebugInformationFormat>
      <Optimization>Disabled</Optimization>
    </ClCompile>
    <Link>
      <TargetMachine>MachineX86</TargetMachine>
      <GenerateDebugInformation>true</GenerateDebugInformation>
      <SubSystem>Windows</SubSystem>
    </Link>
  </ItemDefinitionGroup>
  <ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Release|Win32'">
    <ClCompile>
      <PreprocessorDefinitions>WIN32;NDEBUG;_WINDOWS;%(PreprocessorDefinitions)</PreprocessorDefinitions>
      <RuntimeLibrary>MultiThreadedDLL</RuntimeLibrary>
      <WarningLevel>Level3</WarningLevel>
      <DebugInformationFormat>ProgramDatabase</DebugInformationFormat>
    </ClCompile>
    <Link>
      <TargetMachine>MachineX86</TargetMachine>
      <GenerateDebugInformation>true</GenerateDebugInformation>
      <SubSystem>Windows</SubSystem>
      <EnableCOMDATFolding>true</EnableCOMDATFolding>
      <OptimizeReferences>true</OptimizeReferences>
    </Link>
  </ItemDefinitionGroup>
  <ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Debug|x64'">
    <Link>
      <AdditionalDependencies>python35.lib;kernel32.lib;user32.lib;gdi32.lib;winspool.lib;comdlg32.lib;advapi32.lib;shell32.lib;ole32.lib;oleaut32.lib;uuid.lib;odbc32.lib;odbccp32.lib;%(AdditionalDependencies)</AdditionalDependencies>
    </Link>
  </ItemDefinitionGroup>
  <ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'">
    <Link>
      <AdditionalDependencies>python35.lib;kernel32.lib;user32.lib;gdi32.lib;winspool.lib;comdlg32.lib;advapi32.lib;shell32.lib;ole32.lib;oleaut32.lib;uuid.lib;odbc32.lib;odbccp32.lib;%(AdditionalDependencies)</AdditionalDependencies>
      <AdditionalLibraryDirectories>C:\Python3\libs;</AdditionalLibraryDirectories>
    </Link>
  </ItemDefinitionGroup>
  <ItemGroup>
    <ClCompile Include="call_function.c" />
  </ItemGroup>
  <Import Project="$(VCTargetsPath)\Microsoft.Cpp.targets" />
  <ImportGroup Label="ExtensionTargets">
  </ImportGroup>
</Project>

Be sure all the directories in all the above point to your actual Python3 installation (here it assumes \Python3\ is the installation directory). Then just open up the solution file and build x32 or x64 or whatever you'd like. You'll have to play around a bit to find everything your packages require. For example Numpy requires linking to \Python3\Lib\site-packages\numpy\core\lib\npymath.lib. Any of your own compiled PYD Cython programs can sit directly in \extension_modules\ For my build on Windows I also had to include msvcp140_app.dll in the directory with the EXE (which is strangely \Program Files (x86)\Microsoft Visual Studio 14.0\VC\redist\x64\Microsoft.VC140.CRT\msvcp140.dll renamed, or the \x86\ directory for 32bit).

For all these odd reasons I note, that's why I advise downloading the prebuilt version to get started - at least it will show you how things must be packaged for it to work.

Enjoy.

That looks pretty good. I'm going to need to work with it to see how it goes. I'd like to understand better: How did you compile all of that? What tools & scripts are you using? Can you share any useful references or resources? — Casey, Jul 11 '17 at 00:31
@Casey Do you want the actual Visual Studio solution? For 32bit you'll have to change all your lib / include directories at least. You'll need the embedded Python35.zip and python35.dll and all the things I included in 32bit version under \extension_modules\ instead of what's there as it is all 64bit (i.e. copy all your \site-packages\ 32bit versions to replace my 64bit ones and download 32bit Windows embedded Python 3.5 https://www.python.org/ftp/python/3.5.1/python-3.5.1-embed-win32.zip). — Matt, Jul 11 '17 at 01:02
I'm going to look more at your 64-bit version, which I think will work for me (I'm on 64-bit Windows but 32-bit Python, so maybe it's time to upgrade). More generally, I'm just getting into compiling Python and have only done it using cython so far... and I'd like to learn more about how to get to be able to do what you can do (at least as it pertains to compiling Python). Thanks. — Casey, Jul 11 '17 at 02:28
@Casey All I compiled is this simple example here: https://docs.python.org/3/extending/embedding.html#pure-embedding but the Visual Studio setup was a huge pain to get everything to build using the Python embedded module and extra import libraries. Those items aren't documented well. I would like to provide on a website the Visual Studio project but I'm limited to Dropbox at the moment (unless someone on SO wants to host it)? I also have a version that adds TKinter which was a bit tricky to implement (I've seen postings on SO where people had issues). — Matt, Jul 11 '17 at 13:49
@Casey A 32bit version will work the same way since you're just calling your .py script with the exe file. As long as the computer is 64bit OS which I can't think of any that aren't these days... — Matt, Jul 11 '17 at 16:31

score 0 · Answer 2 · answered Jul 11 '17 at 00:11

0

You should try py2exe, I think that is the easiest way to make a .exe from a python code. I understand what you want, but is easy to create a .exe with py2exe, copy the folder in C and then create a shortcut.

answered Jul 11 '17 at 00:11

S.Moncayo

51
4

I've used py2exe in the past and (after some tweaking for Numpy, matplotlib and such) is worked well. It still requires python to be installed, which I can't guarantee for all of my users. Also, I don't see a precompiled wheel for Python 3.5, so how do you use it given Python 3.5: http://www.lfd.uci.edu/~gohlke/pythonlibs/#py2exe – Casey Jul 11 '17 at 00:23
1

From the first line of the py2exe website: "py2exe is a Python Distutils extension which converts Python scripts into executable Windows programs, able to run __without requiring a Python installation__." – DavidW Jul 11 '17 at 06:55
1

As @DavidW said, .exe files created with py2exe doesn't require a Python installation. I've read that py2exe doesn't works with python 3.5 For that, I used py2exe with Python 2.7 and it's work perfectly. – S.Moncayo Jul 11 '17 at 13:23
DavidW & @S.Moncayo, thanks for the clarification on py2exe. I'm stuck with Python 3.5 because of some of the features in some included modules. I don't see a Windows pre-compiled wheel for py2exe (http://www.lfd.uci.edu/~gohlke/pythonlibs/), which along with DavidW's comment makes me think it's not an option for me. – Casey Jul 11 '17 at 14:04
@Casey My embedded link uses the same libraries from the site you referenced. I'll see if I can zip up a 32bit version when I get home in 8hrs or so – Matt Jul 11 '17 at 14:26

How to use cython_freeze on Windows to build a single executable file?

2 Answers2

Here's the compiled embedded Python 3.5 x64 VS 2015 link w/ NumPy, SciPy, Pandas, Intel MKL:

HOW TO COMPILE YOURSELF