Cannot import Geopandas with PyInstaller executable - despite running fine in the virtual env

Question

When my Python application frozen with PyInstaller attempts to import Geopandas, it stops working.

• Windows 10
• PyInstaller 3.3.1
• Geopandas 0.4

Here is the source code:

print("Hello, StackOverflow")
import geopandas as gpd

Here is the resulting console output of the compiled EXE:

Hello, StackOverflow
Traceback (most recent call last):
  File "application.py", line 3, in <module>
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "d:\documents\projecttwo\publish\harv_venv1\env\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 631, in exec_module
    exec(bytecode, module.__dict__)
  File "site-packages\geopandas\__init__.py", line 9, in <module>
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "d:\documents\projecttwo\publish\harv_venv1\env\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 631, in exec_module
    exec(bytecode, module.__dict__)
  File "site-packages\geopandas\datasets\__init__.py", line 7, in <module>
StopIteration
[6764] Failed to execute script application

This behavior is consistent when I attempt to import Geopandas is much more complex applications, and the console output is constant.

Geopandas is installed correctly within the Python 3.6.3 virtual environment (through PIP, I've tried versions 0.4 and 0.3 as well) and works just fine prior to compilation (ie. python application.py runs successfully).

I have attempted installing both geopandas and pyinstaller from different sources (eg. Gohlke's wheel), with the same results. I have also tried creating an entirely new virtual env from scratch, installing Fiona from Gohlke and geopandas from pip.

I suspect some hidden imports may need to take place. I am fairly new to PyInstaller, so any help would be appreciated.

Answer 1

I received this same error and solved it differently to aorr above.

The error was caused because the geopandas datasets included in the package aren't found by pyinstaller because they are .shp files.

I don't use the geopandas datasets in my project/s so instead of manually including them in my .spec file I just commented out the import geopandas.datasets statement from: File "site-packages\geopandas\__init__.py", line 9, in <module>.

This compiled properly and gave me the expected outputs for my program.

Answer 2

looks like geopandas is aggressively loading their data directory on init. It contains non-python files that are ignored by pyinstaller in your package, so for geopandas to find them when you load it, they must be packaged explicitly.

The "manual" process took me a while to figure out, and I'm using conda as my package manager (if you're not, these edits should still help you). To get this working we need to modify the .spec file that was built when you ran pyinstaller the first time:

# -*- mode: python -*-

import os
from PyInstaller.utils.hooks import collect_data_files # this is very helpful
env_path = os.environ['CONDA_PREFIX']
dlls = os.path.join(env_path, 'DLLs')
bins = os.path.join(env_path, 'Library', 'bin')

paths = [
    os.getcwd(),
    env_path,
    dlls,
    bins,
]

# these binary paths might be different on your installation. 
# modify as needed. 
# caveat emptor
binaries = [
    (os.path.join(bins,'geos.dll'), ''),
    (os.path.join(bins,'geos_c.dll'), ''),
    (os.path.join(bins,'spatialindex_c-64.dll'), ''),
    (os.path.join(bins,'spatialindex-64.dll'),''),
]

hidden_imports = [
    'ctypes',
    'ctypes.util',
    'fiona',
    'gdal',
    'geos',
    'shapely',
    'shapely.geometry',
    'pyproj',
    'rtree',
    'geopandas.datasets',
    'pytest',
    'pandas._libs.tslibs.timedeltas',
]

# other fancy pyinstaller stuff...

a = Analysis(['run_it.py'],
         pathex=paths, # add all your paths
         binaries=binaries, # add the dlls you may need
         datas=collect_data_files('geopandas', subdir='datasets'), #this is the important bit for your particular error message
         hiddenimports=hidden_imports, # double tap
         hookspath=[],
         runtime_hooks=[],
         excludes=excludes,
         win_no_prefer_redirects=False,
         win_private_assemblies=False,
         cipher=block_cipher)
# remaining fancy pyinstaller stuff...

That should collect your missing data directory and put it where your executable can find it.

The "auto" way would be to build a hook-geopandas.py file that does this for you. pyinstaller loads these hooks when you build as a way of saving and sharing these tricks. In fact, there's already a very nice hook file for shapely, one of geopandas dependencies, that you can review here.

------EDIT--------

I am also currently building a project that depends on geopandas and I realized that the fix above is incomplete as of this date (2018-08-23) because of this issue.

In my run_it.py i have included the following test to ensure fiona and gdal are all packaged up correctly into the bundle:

from osgeo import gdal, ogr, osr
from fiona.ogrext import Iterator, ItemsIterator, KeysIterator
from geopandas import GeoDataFrame

This test will probably fail for you unless you're a wizard. This shim worked for me in my .spec file:

_osgeo_pyds = collect_data_files('osgeo', include_py_files=True)

osgeo_pyds = []
for p, lib in _osgeo_pyds:
    if '.pyd' in p:
        osgeo_pyds.append((p, ''))

binaries = osgeo_pyds + [
    # your other binaries
]

a = Analysis(
    # include your kwargs
)

I hope this helps make this answer more complete, and your bundled app do it's geospatial things as expected.