Out of memory with Python but OK with Matlab while displaying the same 2D data as image

Question

I have a 2D array to display as image (it is 500 by 20 000).

Python:

import numpy as np
from matplotlib import pyplot as plt    

spect_data = np.loadtxt('some_data.txt') 

plt.figure(figsize=(12,9))
plt.imshow(spect_data,aspect='auto')
plt.colorbar()
plt.show()

Matlab:

spect_data=load('some_data.txt');
imagesc(spect_data)

Here's the error I get (sorry I wasn't clear about my problem the first time):

Traceback (most recent call last):

File "C:\Users\User\Anaconda\lib\site-packages\IPython\core\formatters.py", line 339, in call return printer(obj)

File "C:\Users\User\Anaconda\lib\site-packages\IPython\core\pylabtools.py", line 228, in png_formatter.for_type(Figure, lambda fig: print_figure(fig, 'png', **kwargs))

File "C:\Users\User\Anaconda\lib\site-packages\IPython\core\pylabtools.py", line 119, in print_figure fig.canvas.print_figure(bytes_io, **kw)

File "C:\Users\User\Anaconda\lib\site-packages\matplotlib\backend_bases.py", line 2180, in print_figure **kwargs)

File "C:\Users\User\Anaconda\lib\site-packages\matplotlib\backends\backend_agg.py", line 527, in print_png FigureCanvasAgg.draw(self)

File "C:\Users\User\Anaconda\lib\site-packages\matplotlib\backends\backend_agg.py", line 474, in draw self.figure.draw(self.renderer)

File "C:\Users\User\Anaconda\lib\site-packages\matplotlib\artist.py", line 61, in draw_wrapper draw(artist, renderer, *args, **kwargs)

File "C:\Users\User\Anaconda\lib\site-packages\matplotlib\figure.py", line 1159, in draw func(*args)

File "C:\Users\User\Anaconda\lib\site-packages\matplotlib\artist.py", line 61, in draw_wrapper draw(artist, renderer, *args, **kwargs)

File "C:\Users\User\Anaconda\lib\site-packages\matplotlib\axes_base.py", line 2324, in draw a.draw(renderer)

File "C:\Users\User\Anaconda\lib\site-packages\matplotlib\artist.py", line 61, in draw_wrapper draw(artist, renderer, *args, **kwargs)

File "C:\Users\User\Anaconda\lib\site-packages\matplotlib\image.py", line 389, in draw im = self.make_image(renderer.get_image_magnification())

File "C:\Users\User\Anaconda\lib\site-packages\matplotlib\image.py", line 624, in make_image transformed_viewLim)

File "C:\Users\User\Anaconda\lib\site-packages\matplotlib\image.py", line 238, in _get_unsampled_image x = (x * 255).astype(np.uint8)

MemoryError

Their functions may not be stored in memory. It may be using [generators](https://wiki.python.org/moin/Generators). — Moon Cheesez, Jul 27 '16 at 23:10
Maybe use a Numpy array `results` instead of a list. [That will save you space](http://stackoverflow.com/a/994010/1586200). — Autonomous, Jul 28 '16 at 04:15
What is the full backtrace of the error message? Without knowing exactly where the memory error is occurring it is impossible to help. — TheBlackCat, Jul 28 '16 at 05:38
Your code runs fine on my linux machine. Maybe it's something to do with anaconda or windows, both of which I don't use. So someone else will be better able to help. Is anaconda up to date? Have you tried a different python distribution? Have you tried `imshow` on other (smaller) data? — Alex, Jul 29 '16 at 12:24

score 2 · Answer 1 · answered Jul 27 '16 at 23:09

2

I'm not sure this will solve your problem, but you seem to have the data in memory several times - as a numpy array, as a list of floats, and as a list of strings.

If you only need the numpy array, you could use

np.loadtxt

or

np.fromfile

if you need more control over how the data is read.

This assumes (you do not specify) that the data is in an ASCII file. For a more specific answer, you should post your code so people can see what you are doing and where the problem might be.

answered Jul 27 '16 at 23:09

Alex

336
3
11

I changed the way the data is loaded (by the way, thanks, it's very useful to know), but still MemoryError... – Sylwia Jul 27 '16 at 23:50
@Sylwia Please update your original post with the new version. – TheBlackCat Jul 28 '16 at 05:37
`imshow` seems to be fine with data of your size. I tried with `a=np.random.normal(size=(500,20000))`, `imshow(a)` with the expected result of seing a plot of the above random data. This is python3.5.2 or 2.7.12 running in ipython. Old laptop with 3GB RAM, but lots of other stuff running as well. What's the exact error you are getting? – Alex Jul 28 '16 at 13:03

Out of memory with Python but OK with Matlab while displaying the same 2D data as image

1 Answers1