Efficient creation of numpy arrays from list comprehension and in general

Question

In my current work, I use Numpy and list comprehensions a lot and in the interest of the best possible performance I have the following questions:

What actually happens behind the scenes if I create a Numpy array as follows?

a = numpy.array( [1,2,3,4] )

My guess is that python first creates an ordinary list containing the values, then uses the list size to allocate a numpy array and afterwards copies the values into this new array. Is this correct, or is the interpreter clever enough to realize that the list is only intermediary and instead copy the values directly?

Similarly, if i wish to create a numpy array from list comprehension using numpy.fromiter():

a = numpy.fromiter( [ x for x in xrange(0,4) ], int )

will this result in an intermediary list of values being created before being fed into fromiter()?

If you're trying to avoid the creation of the list, why `a = numpy.fromiter( [ x for x in xrange(0,4) ], int )` instead of simply `a = numpy.fromiter(xrange(4), int)`? — wim, Jan 17 '13 at 05:21
Just an example (a poor one, i'll admit). The expression could be anything — NielsGM, Jan 17 '13 at 05:23
Note also you have `np.arange` if you need it, but I guess you probably know that already. — wim, Jan 17 '13 at 05:23
The point raised by @wim, is that `numpy.fromiter(list(something), ...` or `numpy.fromiter([something], ...` should _never_ be used! Use always `numpy.fromiter(something, ...` regardless from what `something` is. — Stefano M, Jan 17 '13 at 11:48
I would avoid creating a numpy array from a list comprehension or generator and use `arange` and vectorized manipulations of the resulting array if you possibly can. I'm still fairly new to Python and I was really shocked at the slow performance of a list comprehension in [this answer](http://stackoverflow.com/a/23037258/834521) I just wrote up compared to working directly with `numpy.arange` and vectorized manipulations (and a generator with `from_iter` wasn't much better in my case). — TooTone, Apr 13 '14 at 01:16

score 47 · Accepted Answer · edited Mar 20 '15 at 14:53

47

I believe than answer you are looking for is using generator expressions with numpy.fromiter.

numpy.fromiter((<some_func>(x) for x in <something>),<dtype>,<size of something>)

Generator expressions are lazy - they evaluate the expression when you iterate through them.

Using list comprehensions makes the list, then feeds it into numpy, while generator expressions will yield one at a time.

Python evaluates things inside -> out, like most languages (if not all), so using [<something> for <something_else> in <something_different>] would make the list, then iterate over it.

edited Mar 20 '15 at 14:53

TimStaley

515
4
21

answered Jan 17 '13 at 05:31

Snakes and Coffee

8,747
4
40
60

@JonClements you could apply some function to `x`, and it would be evaluated as needed – Snakes and Coffee Jan 17 '13 at 05:35
In which case that's a valid use-case - but your original example wasn't... :) – Jon Clements Jan 17 '13 at 05:36
@JonClements Yeah. I sort of just copy-pasted the code and changed things. I forgot about that. Thanks! – Snakes and Coffee Jan 17 '13 at 05:37
19

numpy needs to know the size of the generator to allocate memory for it. How does `np.fromiter` handle this? Storing the generated values, and thus defeating the purpose of not generating a list or tuple? Or running the generator twice, one for counting, the other to fill the array? – Jaime Jan 17 '13 at 06:21
1

@Jaime according to the docs, if you specify the size as `count`, then numpy will preallocate the memory - so if you already have it hanging around then you can do that. Otherwise, you are correct - it would have to run the generator, and count the list it made. – Snakes and Coffee Jan 17 '13 at 06:26
3

@Jaime The generator has to be run only once! (Think about side effects, etc. etc. ) I've not read the source code of `fromiter` in `numpy` but for sure `numpy.fromiter(something, int)` is more efficient than `numpy.fromiter(list(something), int)`. `numpy` can use `malloc`/`realloc` for creating an array of objects of `sizeof(int)`. In Cpython a `list` is a mutable collection of heterogeneous objects, so it has a way more complex data structure and allocation strategy. – Stefano M Jan 17 '13 at 11:33
7

From documentation it is pretty clear. *Specify count to improve performance. It allows fromiter to pre-allocate the output array, instead of resizing it on demand.* It does reallocate array when you will hit the capacity. Similar behaviour to `std::vector` in C++ – Cron Merdek Jan 27 '17 at 08:39

wim · Answer 2 · 2017-09-07T18:53:56.200

You could create your own list and experiment with it to shed some light on the situation...

>>> class my_list(list):
...     def __init__(self, arg):
...         print 'spam'
...         super(my_list, self).__init__(arg)
...   def __len__(self):
...       print 'eggs'
...       return super(my_list, self).__len__()
... 
>>> x = my_list([0,1,2,3])
spam
>>> len(x)
eggs
4
>>> import numpy as np
>>> np.array(x)
eggs
eggs
eggs
eggs
array([0, 1, 2, 3])
>>> np.fromiter(x, int)
array([0, 1, 2, 3])
>>> np.array(my_list([0,1,2,3]))
spam
eggs
eggs
eggs
eggs
array([0, 1, 2, 3])

score 2 · Answer 3 · answered Mar 19 '21 at 15:09

To the question in the title, there is now a package called numba which supports numpy array comprehension, which directly constructs the numpy array without intermediate python lists. Unlike numpy.fromiter, it also supports nested comprehension. However, bear in mind that there are some restrictions and performance quirks with numba if you are not familiar with it.

That said, it can be quite fast and efficient, but if you can write it using numpy's vector operations it may be better to keep it simpler.

>>> from timeit import timeit
>>> # using list comprehension
>>> timeit("np.array([i*i for i in range(1000)])", "import numpy as np", number=1000)
2.544344299999999
>>> # using numpy operations
>>> timeit("np.arange(1000) ** 2", "import numpy as np", number=1000)
0.05207519999999022
>>> # using numpy.fromiter
>>> timeit("np.fromiter((i*i for i in range(1000)), dtype=int, count=1000)",
...        "import numpy as np",
...        number=1000)
1.087984500000175
>>> # using numba array comprehension
>>> timeit("squares(1000)",
... """
... import numpy as np
... import numba as nb
... 
... @nb.njit
... def squares(n):
...     return np.array([i*i for i in range(n)])
... 
... 'compile the function'
... squares(10)
... """,
... number=1000)
0.03716940000003888

Efficient creation of numpy arrays from list comprehension and in general

3 Answers3

Linked

Related