How can I transform a list to array quickly in the framework of Mxnet?

Question

I have a list which has 8 elements and all of those elements are arrays whose shape are (3,480,364).Now I want to transform this list to array as (8,3,480,364).When I use the array=nd.array(list) this command,it will takes me a lot of time and sometimes it will send 'out of memory' error.When I try to use this command array=np.stack(list.aixs=0),when I debug the code,it will stay at this step and can't run out the result.So I wonder how can I transform a list to array quickly when I use the Mxnet framework?

Answer 1

Your method of transforming a list of lists into an array is correct, but an 'out of memory' error means you are running out of memory, which would also explain the slowdown.

How to check how much RAM you have left:

on Linux, you can run free -mh in the terminal.

How to check how much memory a variable takes:

The function sys.getsizeof tells you memory size in bytes.

You haven't said what data type your arrays have, but, say, if they're float64, that's 8 bytes per element, so your array of 8 * 3 * 480 * 364 = 4193280 elements should only take up 4193280 * 8 bytes = about 30 Mb. So, unless you have very little RAM left, you probably shouldn't be running out of memory.

So, I'd first check your assumptions: does your list really only have 8 elements, do all the elements have the same shape of (3, 480, 364), what is the data type, do you create this array once or a thousand times? You can also check the size of a list element: sys.getsizeof(list[0]).

Most likely this will clear it up, but what if your array is really just too big for your RAM?

What to do if an array doesn't fit in memory

One solution is to use smaller data type (dtype=np.float32 for floating point, np.int32 or even np.uint8 for small integer numbers). This will sacrifice some precision for floating point calculations.

If almost all elements in the array are zero, consider using a sparse matrix.

For training a neural net, you can use a batch training algorithm and only load data into memory in small batches.