Python uses too much memory when loading a file

Asked Apr 17 '17 at 05:10

Active Apr 17 '17 at 05:29

Viewed 246 times

I have a file that contains about 2000 rows and 100000 columns and the size of the file is about 350 MB. I am trying to load that file into python by doing this

data = pd.read_csv('data.txt', header=None, delim_whitespace=True).as_matrix()

and I just noticed that in my Task manager python.exe uses about 7GB of memory. Note that the line of code shown above is the only task that I did so far besides loading the necessary libraries. What is causing this and how can I fix it ?

Edit: My question is not really related to the thread you indicated since the thread is about handling large files and I am working with a 350MB file and the main purpose of my question is to figure out why it takes 7GB of memory to store a 350MB file.

edited Apr 17 '17 at 05:29

asked Apr 17 '17 at 05:10

Saik

2

Possible duplicate of ["Large data" work flows using pandas](http://stackoverflow.com/questions/14262433/large-data-work-flows-using-pandas) – Fernando Irarrázaval G Apr 17 '17 at 05:16
yeah look at the parameter chunksize – Aklys Apr 17 '17 at 06:11
what is the size without `.as_matrix()` – Grr Apr 17 '17 at 06:20
The memory size without the matrix appears to be the same. I see a huge improvement when I remove the delim_whitespace=True, but this creates an array of 2000 string lines, which is not what I am looking for. – Saik Apr 17 '17 at 07:35
try engine='python' – edyvedy13 Apr 17 '17 at 09:49
adding engine='python' did not fix the problem. It seems like adding that line made the process slower but the memory usage stayed the same. – Saik Apr 17 '17 at 18:18

Python uses too much memory when loading a file

0 Answers0