How can I create the minimum size executable with pyinstaller?

Question

I am on Windows 10, I have anaconda installed but I want to create an executable independently in a new, clean minimal environment using python 3.5. So I did some tests:

TEST1: I created a python script test1.py in the folder testenv with only:

print('Hello World')

Then I created the environment, installed pyinstaller and created the executable

D:\testenv> python -m venv venv_test
...
D:\testenv\venv_test\Scripts>activate.bat
...
(venv_test) D:\testenv>pip install pyinstaller
(venv_test) D:\testenv>pyinstaller --clean -F test1.py

And it creates my test1.exe of about 6 Mb

TEST 2: I modified test1.py as follows:

import pandas as pd
print('Hello World')  

I installed pandas in the environment and created the new executable:

(venv_test) D:\testenv>pip install pandas
(venv_test) D:\testenv>pyinstaller --clean -F test1.py

Ant it creates my test1.exe which is now of 230 Mb!!!

if I run the command

(venv_test) D:\testenv>python -V
Python 3.5.2 :: Anaconda custom (64-bit)

when I am running pyinstaller I get some messages I do not understand, for example:

INFO: site: retargeting to fake-dir 'c:\\users\\username\\appdata\\local\\continuum\\anaconda3\\lib\\site-packages\\PyInstaller\\fake-modules'

Also I am getting messages about matplotlib and other modules that have nothing to do with my code, for example:

INFO:   Matplotlib backend "pdf": added
INFO:   Matplotlib backend "pgf": added
INFO:   Matplotlib backend "ps": added
INFO:   Matplotlib backend "svg": added

I know there are some related questions: Reducing size of pyinstaller exe, size of executable using pyinstaller and numpy but I could not solve the problem and I am afraid I am doing something wrong with respect to anaconda.

So my questions are: what am I doing wrong? can I reduce the size of my executable?

Answer 1

I accepted the answer above but I post here what I did step by step for complete beginners like me who easily get lost.

Before I begin I post my complete test1.py example script with all the modules I actually need. My apologies if it is a bit more complex than the original question but maybe this can help someone.

test1.py looks like this:

import matplotlib 
matplotlib.use('Agg') 
import matplotlib.pyplot as plt
import matplotlib.image as image
import numpy as np
import os.path
import pandas as pd
import re   
    
from matplotlib.ticker import AutoMinorLocator 
from netCDF4 import Dataset
from time import time
from scipy.spatial import distance
from simpledbf import Dbf5
from sys import argv

print('Hello World')

I added matplotlib.use('Agg') (as my actual code is creating figures) Generating a PNG with matplotlib when DISPLAY is undefined

1) Install a new version of python independently from anaconda.

downloaded python from: https://www.python.org/downloads/ installed selecting 'add python to path' and deselecting install launcher for all users (I don't have admin rights) check that I am using the same version from CMD, just writing python I get:

Python 3.6.4 (v3.6.4:d48eceb, Dec 19 2017,
06:04:45) [MSC v.1900 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.

2) Create and activate the environment, from CMD

D:\> mkdir py36envtest
...
D:\py36envtest>python -m venv venv_py36
...
D:\py36envtest\venv_py36\Scripts>activate.bat

3) Install in the environment all the modules needed in the script

Making sure they are compatible to the python version with the command: (from Matplotlib not recognized as a module when importing in Python)

(venv_py36) D:\py36envtest> python -m pip install nameofmodule

NB: in my case I also had to add the option --proxy https://00.000.000.00:0000

for the example I used development version of py installer:

(venv_py36) D:\py36envtest> python -m pip install https://github.com/pyinstaller/pyinstaller/archive/develop.tar.gz

and the modules: pandas, matplolib, simpledbf, scipy, netCDF4. At the end my environment looks like this.

(venv_py36) D:\py36envtest> pip freeze
altgraph==0.15
cycler==0.10.0
future==0.16.0
macholib==1.9
matplotlib==2.1.2
netCDF4==1.3.1
numpy==1.14.0
pandas==0.22.0
pefile==2017.11.5
PyInstaller==3.4.dev0+5f9190544
pyparsing==2.2.0
pypiwin32==220
python-dateutil==2.6.1
pytz==2017.3
scipy==1.0.0
simpledbf==0.2.6
six==1.11.0
style==1.1.0
update==0.0.1

4) Create/modify the .spec file (when you run pyinstaller it creates a .spec file, you can rename).

Initially I got a lot of ImportError: DLL load failed (especially for scipy) and missing module error which I solved thanks to these posts:
What is the recommended way to persist (pickle) custom sklearn pipelines?
and the comment to this answer: Pyinstaller with scipy.signal ImportError: DLL load failed

My inputtest1.spec finally looks like this:

# -*- mode: python -*-
options = [ ('v', None, 'OPTION')]
block_cipher = None


a = Analysis(['test1.py'],
             pathex=['D:\\py36envtest', 'D:\\py36envtest\\venv_py36\\Lib\\site-packages\\scipy\\extra-dll' ],
             binaries=[],
             datas=[],
             hiddenimports=['scipy._lib.messagestream',
                            'pandas._libs.tslibs.timedeltas'],
             hookspath=[],
             runtime_hooks=[],
             excludes=[],
             win_no_prefer_redirects=False,
             win_private_assemblies=False,
             cipher=block_cipher)
pyz = PYZ(a.pure, a.zipped_data,
             cipher=block_cipher)
exe = EXE(pyz,
          a.scripts,
          a.binaries,
          a.zipfiles,
          a.datas,
          name='test1',
          debug=False,
          strip=False,
          upx=True,
          runtime_tmpdir=None,
          console=True )

5) Finally make the executable with the command

(venv_py36) D:\py36envtest>pyinstaller -F --clean inputtest1.spec

my test1.exe is 47.6 Mb, the .exe of the same script created from an anaconda virtual environment is 229 Mb.

I am happy (and if there are more suggestions they are welcome)

Answer 2

The problem is that you should not be using a virtual environment and especially not anaconda. Please download default python 32 bit and use only necessary modules. Then follow the steps provided in the links, this should definitely fix it.

Although you created a virtual env, are you sure your spec file is not linking to old Anaconda entries?

If all this fails, then submit a bug as this is very strange.

Answer 3

I had a similar problem and found a solution.

I used Windows terminal preview. This program allows creation of various virtual environments like Windows Power Shell (btw. Linux Ubuntu too. Also, worth noting: you can have many terminals in this program installed and, even, open a few at once. Very cool stuff).

Inside Windows Power Shell in Windows terminal preview I installed all the necessary libraries (like numpy, pandas,re, etc.), then I opened the path to my file and tried to use this command:

pyinstaller --onefile -w 'filename.py'

...but, the output exe didn't work. For some reason, the console said that there is a lack of one library (which I had installed earlier). I've found the solution in mimic the auto-py-to-exe library. The command used by this GUI is:

pyinstaller --noconfirm --onedir --console "C:/Users/something/filename.py"

And this one works well. I reduced the size of my output exe program from 911MB to 82,9MB !!!

BTW. 911MB was the size of output made by auto-py-to-exe.

I wonder how is it possible that no one yet has created a compressor that reads the code, checks what libraries are part of the code, then putting only them inside the compression. In my case, auto-py-to-exe probably loaded all libraries that I ever installed. That would explain the size of this compressed folder.

Some suggest using https://virtualenv.pypa.io/en/stable/ but in my opinion, this library is very difficult, at least for me.