I have created a csv file which contains a matrix of around 18000x18000. The file itself is a little over 6GB. When I run
simData = np.genfromtxt(FileName, ",");
I get a "MemoryError" after it has run for a few minutes.
Any way around this?
Asked
Active
Viewed 148 times
0
-
Look here, can help. https://stackoverflow.com/questions/48027030/dtype-argument-in-numpy-genfromtxt – kur ag Feb 16 '19 at 04:42
-
How did you create it? – hpaulj Feb 16 '19 at 05:26
-
What kind values does this contain? flats, ints ? – hpaulj Feb 16 '19 at 06:12
-
1I would assume that your RAM is not sufficient. The question has already been answered in this post: https://stackoverflow.com/questions/8956832/python-out-of-memory-on-large-csv-file-numpy. – Lukas Koestler Feb 16 '19 at 11:15
-
contains floats – DaveK Feb 16 '19 at 11:31
-
What's the dtype? Can you fit in RAM a matrix of that type (with zeroes for instance)? If yes, first, create the matrix, then load by batch with your own code and not `numpy.genfromtxt` which can be RAM hungry for large files. – Learning is a mess Feb 16 '19 at 11:32
-
loadtxt is the answer – DaveK Feb 21 '19 at 01:58