2

Version: Python 3.7 (Spyder) -- OS: Windows10 -- System: Core i5 (6th gen) + 16gb RAM

I wrote a program which handles a lot of data. The following structure is used to accomplish this:

Program Description

  • A GUI interface is used as a main function (class). In here an interface pops up, asks the user for input, uses this input the make all kinds of calculation, which are specified in different functions.
  • The first function is an import function, where in a specified (by user) folder is searched for all .wav files and where the data of these are imported. All imported items are appended (numpy.append) to one big array.
  • The big array (for 20 files about 2.000.000.000 datapoints) is used to calculated characteristics of the sound file. The reason it has so many datapoints is because the samplerate of the .wav file is set on 78125 samples/s, which I need for accurate calculations.
  • After the calculations, 2 plots are generated in a specified folder and 2 csv's are also stored in that folder with the requested data.

Problem Statement

  • Running the main function (program) in the spyder environment, works totally fine. It take about 10 minutes to go through all the data and generates the output.

  • Compiling the function to an .exe using PyInstaller, works fine, no errors, everything dependency imported. However when running the program a MemoryError pops up almost immediatly (see image below).

Image: error message from command line when executing the exe file

Tried solutions

  • Running the python script via CLI, gives the same error

  • Running the .exe program with only 2 files to import, works all file, but incredibly slow (much slower than executed via spyder)

Questions

  • Why has spyder enough memory to process all data with no problems, but when executing the .py via command line or when executing the .exe file, there is always a memory error?
  • Why does the .exe or the .py via CL run slower than in the spyder IDE?

Goal

This program should be able to process noise data on every laptop in the company (also 8gb ram sometimes). So I want to find a way to let the program allocate all available RAM on the used machine.

Thanks in advance, for any help!

Cedric Dhont
  • 41
  • 4
  • 1
    Which python interpreter is Spyder using? Is it the same as the one you're accessing via command line? This almost sounds like a 32 bit versus 64 bit issue. – Axe319 Jan 29 '20 at 12:03
  • Should the command prompt not use automatically the interpreter in which the program is written in? As i need it to be a standalone program, it can not be dependent on an installation of python. Or at least, that was what I hoped for! – Cedric Dhont Jan 29 '20 at 12:41
  • In my spyder console it states the following: 'Python 3.7.4 (default, Aug 9 2019, 18:34:13) [MSC v.1915 64 bit (AMD64)]'. I use spyder via Anaconda. However my python interpreter that I installed is Python 3.8.0 (32-bit). Yet my computer is a 64bit operating system. Does this mean I nead to reinstall a 64 bit version of the pyhon interpreter? – Cedric Dhont Jan 29 '20 at 12:45
  • I now installed python 3.8.1 which is a 64bit version. I tried running my .exe program gain, but I got the same error message: MemoryError: `Unable to allocate 1.26gB for an array with...`. I checked the version of python in the command prompt via python --version, an the reply was python 3.8.1. Which indicates it was updated to the 64 bit version right? – Cedric Dhont Jan 29 '20 at 12:57
  • 1
    Assuming you are running your script like `python your_script.py`, you can tell which version it is executing under from the command line by typing `python` and hitting enter. This will put you into the interpreter and you should see something like `Python 3.7.4 (tags/v3.7.4:e09359112e, Jul 8 2019, 19:29:22) [MSC v.1916 32 bit (Intel)] on win32`. If it's 32 bit, you're running the script under 32 bit and so on. – Axe319 Jan 29 '20 at 13:05
  • 1
    And no, the command prompt does not know anything about your script. it simply uses the version of python that the `python` path environmental variable is pointing towards. – Axe319 Jan 29 '20 at 13:08
  • If I follow your instruction, the command prompt returns this: `Python 3.8.1 (tags/v3.8.1:1b293b6, Dec 18 2019, 23:11:46) [MSC v.1916 64 bit (AMD64)]` on win32´´. Which I find very confusing, as I run now a 64bit python version but on a win32? How can I change it to a win64 application? + More importantly: how do I make the program automatically run on the 64bit, on another computer without python installed? – Cedric Dhont Jan 29 '20 at 13:09
  • 1
    `win32` is simply saying that the platform is windows. 64 bit means that you are running the 64 bit version. See here https://stackoverflow.com/questions/17757819/what-does-the-python-version-line-mean . As far as targeting 64 bit, someone can correct me if I'm wrong but I believe pyinstaller will use the python version the executable is built on. So if you `pip install pyinstaller` on your new 64 bit version it should create a 64 bit executable. – Axe319 Jan 29 '20 at 13:28
  • 1
    YES!!! After installing pyinstaller (and all packages for that matter) again using the new python 64bit, the .exe file runs smoothly and without any issues. It runs in about 7minutes through all files, which is the same as running from the spyder IDE. Thank you very much, this was really helpful!! – Cedric Dhont Jan 29 '20 at 13:54
  • 1
    No problem! If you want to add an answer for others that might come across this, feel free. – Axe319 Jan 29 '20 at 13:58
  • 1
    Done, thanks again – Cedric Dhont Jan 29 '20 at 14:13

1 Answers1

2

Meanwhile I have found the answer to my question thanks to Axe319:

The Spyder IDE was running on a 64bit version of python, making the program run smoothly and without any issues. Nevertheless my python interpreter was still a 32bit version of python.

Steps taken to solve the problem:

  • uninstall python 32bit version
  • install python 64bit version
  • install all used packages again using pip install -packages-
  • install PyInstaller again using pip install pyinstaller
  • Compile the program to .exe using PyInstaller

After this all seems to be working!

Cedric Dhont
  • 41
  • 4