Performance of numpy.insert dependant from array size - workaround?

Question

Using the following code, I get the impression that the insert into a numpy array is dependant from the array size.

Are there any numpy based workarounds for this performance limit (or also non numpy based)?

if True:
    import numpy as np
    import datetime
    import timeit
myArray = np.empty((0, 2), dtype='object')
myString = "myArray = np.insert(myArray, myArray.shape[0], [[ds, runner]], axis=0)"
runner = 1
ds = datetime.datetime.utcfromtimestamp(runner)
% timeit myString
19.3 ns ± 0.715 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)
for runner in range(30_000):
    ds = datetime.datetime.utcfromtimestamp(runner)
    myArray = np.insert(myArray, myArray.shape[0], [[ds, runner]], axis=0)
print("len(myArray):", len(myArray))
% timeit myString
len(myArray): 30000
38.1 ns ± 1.1 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)

Instead of starting with a small array and inserting chunks into it, did you consider creating the array in its **full and final size** and then **assigning** values into different parts of that array? Whenever your logic permits, that is always a faster option, compared to multiple inserts. In the end, you could even perform a single numpy.delete operation if you're still left with a single unwanted part. — fountainhead, Mar 01 '19 at 14:16

score 4 · Answer 1 · answered Mar 01 '19 at 14:10

4

This has to do with the way numpy works. For each insert operation, it takes the whole array and stores it in a new place. I would recommend using list append and convert it then to a numpy array. Maybe duplicate of this question

answered Mar 01 '19 at 14:10

anki

765
5
14

score 3 · Accepted Answer · answered Mar 01 '19 at 20:44

Your approach:

In [18]: arr = np.array([])                                                     
In [19]: for i in range(1000): 
    ...:     arr = np.insert(arr, arr.shape[0],[1,2,3]) 
    ...:                                                                        
In [20]: arr.shape                                                              
Out[20]: (3000,)

In [21]: %%timeit  
    ...: arr = np.array([]) 
    ...: for i in range(1000): 
    ...:     arr = np.insert(arr, arr.shape[0],[1,2,3]) 
    ...:                                                                        
31.9 ms ± 194 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

Compare that with concatenate:

In [22]: %%timeit  
    ...: arr = np.array([]) 
    ...: for i in range(1000): 
    ...:     arr = np.concatenate((arr, [1,2,3])) 
    ...:                                                        
5.49 ms ± 20.6 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

and with a list extend:

In [23]: %%timeit  
    ...: alist = [] 
    ...: for i in range(1000): 
    ...:     alist.extend([1,2,3]) 
    ...: arr = np.array(alist)                                                                        
384 µs ± 13.2 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

We discourage the use of concatenate (or np.append) because it is slow, and can be hard to initialize. List append, or extend, is faster. Your use of insert is even worse than concatenate.

concatenate makes a whole new array each time. insert does so as well, but because it's designed to put the new values anywhere in the original, it is much more complicated, and hence slower. Look at its code if you don't believe me.

lists are designed for growth; new items are added via a simple object (pointer) insertion into a buffer that has growth growth. That is, the growth takes occurs in-place.

Insertion into a full array is also pretty good:

In [27]: %%timeit  
    ...: arr = np.zeros((1000,3),int) 
    ...: for i in range(1000): 
    ...:     arr[i,:] = [1,2,3] 
    ...: arr = arr.ravel()                                                                      
1.69 ms ± 9.47 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

Performance of numpy.insert dependant from array size - workaround?

2 Answers2