5

I have been trying to use PyInstaller to distribute my program as a bundled .exe file to my colleagues. The program, which analyses text data, runs perfectly from my CMD. I am using PyInstaller 3.6, Python 3.7, Windows 10, and Anaconda3 as my python environment. The program has quite a few dependencies including nltk, gensim, wordcloud, sklearn, matplotlib, mpld3, seaborn, pandas, numpy, xlsxwriter and a few standard libraries. It is also quite a long program, ~2000 lines.

The Problem: So far, I've successfully built the .exe file (as a one-folder and one-file bundle). To test whether the .exe file works, I run it from my CMD. All is going well (my imports and functions run OK, I am prompted to enter the name of the Excel file containing text data, the text is cleaned etc) but it abruptly terminates, without any warning or error message, on a line that uses Gensim's SparseTermSimilarityMatrix function. I experimented by commenting out this line and it terminates on the very next line, again without any message.

I am building the .exe file using a .spec file (so that I can add data files to my bundle and several hidden imports to overcome Module Not Found Errors). The only indication I have as to what is causing this problem are warnings about missing DLLs during build-time:

108890 INFO: Looking for dynamic libraries
109047 WARNING: lib not found: msmpi.dll dependency of C:\Users\username\anaconda3\envs\pyexe\Library\bin\mkl_blacs_msmpi_lp64.dll
109293 WARNING: lib not found: impi.dll dependency of C:\Users\username\anaconda3\envs\pyexe\Library\bin\mkl_blacs_intelmpi_lp64.dll
109304 WARNING: lib not found: impi.dll dependency of C:\Users\username\anaconda3\envs\pyexe\Library\bin\mkl_blacs_intelmpi_ilp64.dll
109704 WARNING: lib not found: mpich2mpi.dll dependency of C:\Users\username\anaconda3\envs\pyexe\Library\bin\mkl_blacs_mpich2_lp64.dll
109754 WARNING: lib not found: pgc14.dll dependency of C:\Users\username\anaconda3\envs\pyexe\Library\bin\mkl_pgi_thread.dll
109757 WARNING: lib not found: pgf90rtl.dll dependency of C:\Users\username\anaconda3\envs\pyexe\Library\bin\mkl_pgi_thread.dll
109761 WARNING: lib not found: pgf90.dll dependency of C:\Users\username\anaconda3\envs\pyexe\Library\bin\mkl_pgi_thread.dll
110120 WARNING: lib not found: msmpi.dll dependency of C:\Users\username\anaconda3\envs\pyexe\Library\bin\mkl_blacs_msmpi_ilp64.dll
110164 WARNING: lib not found: mpich2mpi.dll dependency of C:\Users\username\anaconda3\envs\pyexe\Library\bin\mkl_blacs_mpich2_ilp64.dll
112452 WARNING: lib not found: icuuc53.dll dependency of C:\Users\username\anaconda3\envs\pyexe\Scripts\Qt5Core.dll
112459 WARNING: lib not found: icuin53.dll dependency of C:\Users\username\anaconda3\envs\pyexe\Scripts\Qt5Core.dll

I would be very grateful if someone could explain:

1. Why a PyInstaller-built .exe file might terminate early without an error message - even in CMD?

2. How to address missing DLLs (which indeed I cannot find on my computer)?

UPDATE: the .exe file successfully runs from the CMD if I first activate my conda environment! conda activate [myenv] However, this defeats the purpose of a standalone distribution.

jmoyes
  • 181
  • 9
  • I should also add: in my warn.txt file, I have a slew of warnings about missing modules. However, from my research it seems that this is commonly the case and usually benign. – jmoyes Jun 28 '20 at 16:53
  • 1
    From my investigations, the problem seemed to be in the threads- my first clue was that they were all being handled nice and sequentially instead of the usual jumble of output, so I suspect they were all running on a single core. I didn't find a reason why it silently exited. I had the same result as your UPDATE- it would run if I activated my conda environment, and, stranger still, even if I subsequently deactivated it! There was no difference in paths that mattered, but some of the environment must persist. – rolinger Jul 01 '20 at 16:05

1 Answers1

10

After realising the .exe file successfully runs from my CMD if I first activate my anaconda environment, here was my original solution, which was only suitable for a one-folder bundle:

From the root directory of my anaconda environment, I searched for all "dll" files (which returns a lot). I copied all DLL files returned by the search to the "dist" folder of my bundle, skipping duplicates.

The .exe file no longer terminated early and worked perfectly!

It seems that the warnings I received about missing DLLs during build-time were misleading - the DLL files quoted are still not on my computer.

SOLUTION UPDATE: After manually copying all DLL files from my anaconda environment to the dist folder of my bundle, I experimented by removing each DLL file I manually added one by one and testing whether the .exe file still worked or terminated early as before. It came down to just one DLL file: libiomp5md.dll - this was, quite literally, the missing link! Therefore, my recommendation:

  1. search for "libiomp5md.dll" within your anaconda environment and copy it to the directory containing your .spec file
  2. add it as a data file in your .spec file and specify that it should be stored in root directory of your bundle: datas=[('libiomp5md.dll', '.')]
  3. Build your executable using the .spec file (either as one-folder or one-file bundle)!

I hope this helps anyone facing similar DLL-related issues.

jmoyes
  • 181
  • 9
  • That is brilliant @jmoyes and couldn't have been more perfectly timed. I've just spent a couple of days fighting with pyinstaller and came down to the same problem. Thanks for taking the time to write up the question and answer, libiomp5md.dll worked for me as well. – rolinger Jul 01 '20 at 16:02
  • @rolinger great to hear this helped you too! – jmoyes Jul 07 '20 at 20:57
  • 1
    I had a similar problem, with a few differences: the problem was in `sklearn.cluster`'s `KMeans.fit` method. Also, the bug didn't occur in the same computer I was building the exe, I only had this problem when ran the exe in a different computer. Adding the `libiomp5md.dll` solved this issue for me as well. – Rodrigo Bonadia Oct 20 '20 at 17:14
  • Awsome solution, was stuck on exactly this for hours. For me it needed both `libiomp5md.dll` and `mkl_intel_thread.dll` but the same solution applied. – iMS44 Jan 19 '21 at 08:07
  • Instead of copying its dll library you can put intel-openmp into hidden imports. – mhrvth Nov 19 '21 at 16:48
  • ... or collect all dependencies of whatever needs the OpenMP automatically: `from PyInstaller.utils.hooks import collect_submodules` `from PyInstaller.utils.hooks importcollect_data_files` `hidden_imports.extend(collect_submodules('scipy'))` `datas.extend(collect_data_files('scipy'))` – mhrvth Nov 19 '21 at 16:57