PyInstaller and Pandas

Question

I have a fairly simple Python module that I am trying to compile into a Windows .exe file. In my script I am using the wxPython and Pandas libraries. The PyInstaller .exe file that is generated only works/opens when the Pandas library is excluded from my module.

I am getting the same issue whether I use --onefile or --onedir in PyInstaller. I found online that the "new" version of PyInstaller (2.1) should have taken care of this bug. Does anyone have any ideas on what to do?

PyInstaller: version 2.1
pandas: version 0.15.2
Python: version 2.7

My post was delete by [ChrisF♦](https://stackoverflow.com/users/59303/chrisf) , hope this link helps anyone in search of answer for this in detail [Pyinstaller created exe file can not load a keras nn model](https://stackoverflow.com/questions/45830206/pyinstaller-created-exe-file-can-not-load-a-keras-nn-model#answer-45842864) — Santhosh Dhaipule Chandrakanth, Jul 18 '19 at 13:49

score 21 · Answer 1 · edited May 23 '17 at 12:23

I ran into the same issue. I boiled it down to a simple script like this Hello.py:

import pandas
print "hello world, pandas was imported successfully!"

To get pandas to import at run-time correctly I had to modify the Hello.spec to the following:

# -*- mode: python -*-

block_cipher = None

def get_pandas_path():
    import pandas
    pandas_path = pandas.__path__[0]
    return pandas_path

a = Analysis(['Hello.py'],
         pathex=['C:\\ScriptsThatRequirePandas'],
         binaries=None,
         datas=None,
         hiddenimports=[],
         hookspath=None,
         runtime_hooks=None,
         excludes=None,
         win_no_prefer_redirects=None,
         win_private_assemblies=None,
         cipher=block_cipher)

dict_tree = Tree(get_pandas_path(), prefix='pandas', excludes=["*.pyc"])
a.datas += dict_tree
a.binaries = filter(lambda x: 'pandas' not in x[0], a.binaries)

pyz = PYZ(a.pure, a.zipped_data,
         cipher=block_cipher)
exe = EXE(pyz,
      a.scripts,
      exclude_binaries=True,
      name='Hello',
      debug=False,
      strip=None,
      upx=True,
      console=True )
scoll = COLLECT(exe,
           a.binaries,
           a.zipfiles,
           a.datas,
           strip=None,
           upx=True,
           name='Hello')

I then ran:

$pyinstaller Hello.spec --onefile

from the command prompt and got the 'hello world' message I expected. I still don't completely understand why this is necessary. I have a custom build of pandas - which is hooked into the MKL libraries - but it isn't clear to me that this is causing the run failure.

This is similar to the answer here: Pyinstaller not correclty importing pycripto... sometimes

@ physicistintheory -- After using your suggestion, my program worked great for me with console=True, but it is not working when the console=False. But, when I say True I get the black command window at the end along with my wanted exe window which is not required. When I try deleting the black window my desired window also get closed. Can you please suggest me! — ali, Dec 12 '18 at 08:39

score 13 · Answer 2 · answered Nov 26 '17 at 22:06

13

I had a similar issue with pyinstaller version 3.3. The solution was that there was a missing hiddenimport hook as described here

I created a new file under Pyinstaller/hooks/ called hook-pandas.py and put the content as described in this commit here and reinstalled pyinstaller manually via python setup.py install in the Pyinstaller directory.

The issue did not recur when I created exe from my pandas script with pyinstaller using the --onefile option.

answered Nov 26 '17 at 22:06

John

768
1
13
21

3

This worked for me, though I didn't have to reinstall Pyinstaller. – kaisquared Mar 05 '18 at 02:52
This answer simplifies it all to a bunch of command lines : https://stackoverflow.com/a/48846546/4355695 – Nikhil VJ Apr 24 '18 at 20:51

score 5 · Answer 3 · answered Jul 11 '18 at 21:17

5

Just as another solution, adding --hidden-import=pandas._libs.tslibs.timedelta or whatever the module is missing to the pyinstaller command also works.

This may be helpful if you don't want to touch the source of pyinstaller.

answered Jul 11 '18 at 21:17

Alto.Clef

141
2
4

score 3 · Answer 4 · answered Jul 23 '20 at 08:57

I had exactly the same problem, and I found another solution (the only one that worked for me actually) :

I followed pretty much this : https://medium.com/@liron92/pyinstaller-with-pandas-problems-solutions-and-workflow-with-code-examples-c72973e1e23f

Except for the fact that I run my virtual environnement on Anaconda.

Quick step-by-step sollution

Before I start : these are the steps that I followed for my particuliar case, you might want to adapt a little bit depending on your situation.

1) Create and configure your virtual environnement

I used Anaconda to create my env :

conda create --name myenv

Then I installed all the module I needed :

conda install -n myenv pandas
conda install -n myenv -c conda-forge python-docx
etc.

2) Activate your environnement and switch to project path

On Anaconda Prompt :

conda activate myenv
cd path/to/your/project/folder

3) Create and modify your *.spec file

Still on the same Anaconda Prompt window :

pyi-makespec project.py

Then open your project.spec file, it will looks like that:

# -*- mode: python ; coding: utf-8 -*-

block_cipher = None


a = Analysis(['project.py'],
             pathex=['path/to/your/project/folder'],
             binaries=[],
             datas=[],
             hiddenimports=[],
             hookspath=[],
             runtime_hooks=[],
             excludes=[],
             win_no_prefer_redirects=False,
             win_private_assemblies=False,
             cipher=block_cipher,
             noarchive=False)
pyz = PYZ(a.pure, a.zipped_data,
             cipher=block_cipher)
exe = EXE(pyz,
          a.scripts,
          [],
          exclude_binaries=True,
          name='main',
          debug=False,
          bootloader_ignore_signals=False,
          strip=False,
          upx=True,
          console=False )
coll = COLLECT(exe,
               a.binaries,
               a.zipfiles,
               a.datas,
               strip=False,
               upx=True,
               upx_exclude=[],
               name='main')

You just modify hiddenimports = [] and add all the implicit import (which include pandas). In my case, I was also using Tkinter, so I specified :

hiddenimports=['pandas', 'tkinter']

4) Finally run pyinstaller

On the same Anaconda Prompt window (environnement activated, on your project directory) :

pyinstaller main.spec

And then you're done !

Disclaimer

I see everywhere people telling you that you should use --onefile when you compile a Python project with Pyinstaller, I honestly think you shouldn't : it makes the *.exe way slower. Maybe I'm missing something with this so please if you use it explain to me.

Specs

Windows 10
Anaconda 4.8.2
Python 3.7.6
Pandas 1.0.5

Bashash911 · Answer 5 · 2021-11-24T15:48:24.097

if anyone here is using python 3.10 :

you will probably encounter "IndexError: tuple index out of range" error.

to fix it:

I found a solution to this problem. This is a python 3.10 bug, and after applying the fix everything related to pyinstaller started working properly.

" so basically you have to go to the folder "Python310\Lib" and edit the file 'dis.py'. In the 'dis.py' file you have to find this "def _unpack_opargs" and inside the else statement write a new line with this: "extended_arg = 0".

I did something like that:

    else:
        arg = None
        extended_arg = 0 
    yield (i, op, arg)

and everything is working fine now. " https://github.com/pyinstaller/pyinstaller/issues/6301#issuecomment-974927257

score 0 · Answer 6 · answered Nov 17 '18 at 01:27

I solved the same problem by using a hook file in the project directory (per pyinstaller document), hook-pandas.py

hiddenimports = [
    'pandas._libs.tslibs.timedeltas',
    'pandas._libs.tslibs.nattype',
    'pandas._libs.tslibs.np_datetime',
    'pandas._libs.skiplist',
]

then adding one line in the spec file:

...
a = Analysis([...
hookspath=['.'],
...],
...

I tried to include hiddenimports=[..., 'pandas', ...] in the spec file, somehow that didn't work out as expected.

score 0 · Answer 7 · answered Apr 06 '20 at 14:17

0

With python version=3.8 and pyinstaller=3.6, no need to custom the pyinstaller or add pandas hook, the hook-pandas.py already exists in Lib\site-packages\PyInstaller\hooks and everything works fine.

answered Apr 06 '20 at 14:17

vpa

341
1
2
8

Did not work for me when the .exe was created on Windows 7 but I tried running it on Windows 10. Pyinstaller 3.6, Python 3.8.2 – Charalamm May 26 '20 at 20:26
Do not work for me with python 3.8.3 and pyinstaller 3.6 – EBH Jun 03 '20 at 13:21

score 0 · Answer 8 · answered Jun 25 '21 at 09:12

For anyone looking to convert Python scripts with pandas involved to exe in 2021

I struggled a lot with pyinstaller but got perfect results with Nuitka. Perks:

low file size (500kB)
quick build and execution time (similar to Python execution time)
and best of all: pandas support out-of-the-box without virtual envs!

Just install nuitka

pip install nuitka

and build the exe

nuitka hello.py

You can even use upx afterwards to compress further!

upx hello.exe

In my case reduced file size to 1/3, just ~150kB .exe file size for a simple pandas script reading and saving an excel-file.

See my blog post for further details.

score 0 · Answer 9 · answered Apr 11 '22 at 16:28

0

This issue has been solved in pyinstaller 4.10

answered Apr 11 '22 at 16:28

shivam sapdhare

1

Andrew Ward · Answer 10 · 2023-03-13T22:39:53.447

My solution found in 2023 March:

ADD pyinstaller to the .venv package list

Background:

Environment:
    MacOS Ventura (13.2.1)
    Developed in PyCharm @022.3.2 (Professional Edition) 
    Using pipenv version 2023.2.18 (brew installation)
    pandas 1.5.3
    pyinstaller 5.9.0

The solution I found was to make SURE I had installed pyinstaller as a package into Pycharm's Settings/Project/Python Interpreter/package list --- added pyinstaller

And now everything WORKS fine.